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FOREWORD 



This research was conducted under Project 2801, Information System Design 
Methodology; Task 280104, Computer-Aided Instructiorj. Techniques. The 
objective of this task is to develop design principles and specifications 
for automated systems which support training and performance-aiding. Such 
systems can be built into Air Force computer-based information systems and 
used in a time-sharing mode for on-site training and performance-aiding 
of system personnel; or they can be used in Air Force resident schools for 
training the student population. 

This report is one in a series supporting Task 280104. It is the second 
of two reports on a systematized technique for formatting and sequencing 
training and performance-aiding information. The study was aimed primarily 
at a specification methodology for dynamic Information displays sequenced 
by a computer. The principles Involved are also applicable to co’~'?entional 
hard copy Information display, e.g,, military manuals. 

This design study was carried out between November 1969 and February 197]. 
under Contract No. Fl9628-70-C"0103 with Information Resources, Inc, 
Cambridge, Massachusetts. Project staff members and their areas of respon- 
sibility were; Robert E. Horn, who directed the project and undertook docu- 
mentation and classification functions; Richard A. Romaii^ who was primarily 
responsible for the design of the executive control program (the sequenca 
generator) and the command language; Elizabeth H. Nicol, general project 
consultant, who was engaged in planning, analysis, and evaluative functions 
and who wrote the final report; Margaret P, Razar, who served as project 
coordinator and who wrote extensive systerr documentation. 



The Air Force Task Scientist and Contract Monitor was Sylvia R. Mayer, 



Publication of this report does not constitute Air Force approval of the 
report's findings or conclusions. It is published for the exchange of 
information and stimulation of ideas. 
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ABSTRACT 



A new conception of computer-based Instructional systems is presented in this 
design of a system that can deliver Individualized information sequences not 
only to learners and trainees, but to reference workers, reviewers, browsers 
and the like. Underlying the system is a flexible data base, organized 
into labelled, movable information blocks according to the principles of 
Information Mapping- -a system for categorizing and displaying information. 
This report is itself written in modified Information-Mapping style. A sig- 
nificant feature of this computerized infonnation service is that the control 
over information selection and arrangement can be assigned entirely to the 
user, entirely to the system, or to both in one of many possible patterns of 
shared responsibility. When the S)Jtem takes part in infonnation-sequencing 
decisions, its many mechanisms for Individualizing come into play. The 
executive program consults short-term and long-term data about the individual 
his objectives, capabilities, interests and present status before it selects 
and arranges blocks from the data base to display for him. Evaluation and 
feedback provisions are also individualized. The system's capability for 
controlling conditions and recording user-system interactions wi 1 1 make it, a 
valuable force in research on individualization in training and education. 

The development of this complex design was facilitated by a Documentation- 
Updating System that produced system documents in Information-Mapped fomi 
and kept them up to date throughout the project. 
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CHAPTER 1 INTRODUCTION TO THE PROJECT 
OVERVIEW OF THIS DOCUMENT 



Intr’oduction This document describes the design of a computer- control led 

information system whose specially organized datu. base serves 
a variety of users' purposes: initial learning, reference, 
reviewing, browsiiig, and briefing. 

The data base consists of information segments organized 
according to Information Mapping, a system of rules and policies 
for classifying and formatting information into separate labelled 
modules. 'Iliese information blocks can be drawn from the data 
base, arranged and displayed in a form best suited to the purpose 
of the individual client . 

^ TTils document describes how the system design accomplishes the 

objective of delivering information sequences to the individual 
taking into account his capabilities and previous experience. 



The First 
Chapter 



The Second 
Chapter 



The Central 
Chapters 



Documenta- 

tion- 

Updating 

System 



TTie Last 
Chapters 



An information system that is to serve multiple purposes from a 
coimnon information base must rely on a variety of concepts and 
procedures to accomplish its functions. This first chapter will 
introduce the main features of the system. 



Information Mapping is one of the key concepts in this system 
design. Here we tell what it is, how it developed from research 
results, educational practice, graphics, and so on. Its applica- 
tion to computer data bases requires some new procedures, which 
are outlined here. 



liie mechanisms and procedures that were designed to select and 
deliver appropriate information sequences are described in the 
next three chapters. How the system makes an individualized 
response to members of each user class is es^lained, here and 
illustrated with fictitious case histories. 



Hie data and information being developed about the Learning- 
Reference System were documented from the beginning and were 
maintained in an updated state throughout the project. The 
methods for producing documents in Information Mapped form and 
the explicit updating procedures are described. 



The final chapters sununarize the description of the total system, 
some ways of evaluating it, and some future applications. 



THE CONTEXT OF THE NEW SYSTEM 



IVhen a new developnient ©merges , we are usually curious to know 
where it fits into the current scene. 

The system we are _ introducing here is a new kind of computer- 
assisted instructiona It serves reference uses, reviewing, 
browsing and briefing, too, but it is within the context of 
computer-aided training and instructional systems that its 
similarities and differences may best be identified. 



Back in the early 1950's, one of the impressive selling points 
of teaching machines was the fact that they let the individual 
proceed at his own pace , Ihrough the heyday of teaching 
machines on into the development of more versatile computer- 
based systems, the search for new ways of adapting instructional 
sequences to the individual user has continued. 

In spite of undoubted gains, the goal of adiievlng individual- 
ized instruction has been advanced only slowly. The main 
progress has been in the area of adjusting instruction to 
short-term informafion^ about the user-- a wrong answer, for 
instance, causes him to be shunted off to a corrective sequence, 
while consecutive correct responses may cause him to by-pass * 
some materials. The long-term history of the user seldom 
enters into information sequencing decisions. 

The system we are reporting is designed with the capability to 
respond to the individual by basing its sequencing decisions on 
faQfh permanent and changing information about the user--for 
example: 

his capabilities, attitudes. Interests, objectives 

. his academic training and history 

. his performance and patterns of behavior within the 
learning system 

. his personal wishes and preferences. 

Thus the system has the capability of responding to the long- 
term as well as short-term history of the user. The extent to 
which this capability is us^d depends upon whether the course 
designer will specify how individual differences are to be 
served by differences in infonnation sequencing. It also 
depends upon whether the individual user is guided by the 

system or controls his own interaction with the information 
base. 

continued on next page 
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THE CONTEXT OF THE NEW SYSTEM, continued 



Control of M important difforence between this system and others lies in 
Instructional the variations in degrees of control that this system can 
Conditions exercise over the sequencing of informations 

In the majority of CAI systems, the path of the learner is 
almost totally determined by the computer program. In a few 
other systems, the so-called **ad lib** systems*, it is the user 

who determines what information is called up from the data 
base. 



Our Learning— Reference System is designed to operate not only 
under either one of these extremes of total user control or 
total system control, but also at any one o^ numerous pT)ints 
in between. In other words, the learner can take charge of 
all sequencing decisions, he can share some with the system, 

or he can be totally dependent on the system for information 
selection and display. 

This range of possible controls adds another set of dimensions 
for individualizing instruction, but, most importantly at this 
l^erfect stage of our knowledge, it allows experimental control 
of variations in learning conditions so that r es e ar ch may re fin 
the prescriptions that link personal data, information sequences, 
and learning results. 

Such an outcome is realizable because the system has record- 
keeping advantages that permit study of long-term effects of 
sequencing strategies. 



After the capabilities of this Learning-Reference System have 
been described in this report, we shall return, in the final 
chapter, to some of the topics raised here. 



* Bryan, G.L. ’’Current ONR Research Efforts Involving 
Computerized Instruction,” Proc, of the Conference on 
Applications of Computers to Training (ACT), Washington, D.C.. 

1970. S3 
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THE ORIGINS 



Introduction 



Basic 

Objective of 
This Project 

The Learning- 

Reference 

System 



The Design 
Process 



Comment 



THE PROJECT 



The idea of desiijning a computer-controlled system that would 
serve many users from a common information pool originated in a 
previous research and development effort to design more effective 
materials for both learning and reference users* 

From that work, Infonnation Mapping emerged as a system of rules 
for identifying, categorizing and interrelating the information 
needed for learning-reference purposes . 

IVhen applied to my subject matter, Infonnation Mapping yields 
a network of labelled and classified information blocks. From 
these blocks, materials in book form can be produced for self- 
instruction and reference purposes. 

It was no large step to conjecture that if a computer data base 
were organized by Information Mapping rules, it might serve the 
information needs of a wider range of users. The separate blocks 
could be drawn from the conmion pool as needed and then assembled 
into sequences suitable tor initial learning, reference work, 
browsing, reviewing and the like. 

In order to accomplish such a service, the system must be able 
to evaluate and respond to the needs of the various users and 
then to deliver information sequences that are appropriate. 

The objective of this project was to plan a computer infoTTnation 
system that selects blocks from an Infomatlon-Mapped data base 
and assembles them into displays of infoination suited to the 
needs of the individual user. 



Although we conceive of this system as providing information 
services to a variety of users, we refer to it as the Leamlng- 
Reference System because we expect the main traffic to be from 
those with initial learning and reference needs. Nevertheless 
it is also designed to serve those who come for reviewing, brief- 
ing or browsing. 

The project began with general ideas about the nature of the 
system, then the characteristics desirable in such a system were 
fonnulated in detail, ITie main work of the project consisted in 
developing a design for the structures and programs that would 
make the system a reality. 

This report is the description of the system design that evolved. 

Before taking up the process of defining the system, we first 
say more about Infonnation Mapping Itself for it plays a funda- 
mental role in the system. 
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INFORMATION MAPPING: ITS BACKGROUND 



Description 



Basic Ainis 



Ir formation Mapping is a set of rules and procedures for writing, 
organizing and displaying information about a subject matter. 
According to a unique classification scheme, information is cate- 
gorized and collected into labelled blocks. The order and format 
in which these blocks are displayed are geared toward making 
learning easier and information retrieval more efficient. 

The system has been applied to the production of books for 
learning and reference and now in this project it is applied to 
the organization of multipurpose data bases for computerized 
information systems. 



Research and development on Information Mapping have been concerned 
primarily with these objectives: 

. to make learning, reviewing, reference work and browsing 

easier and quicker than in conventional texts and in computer- 
aided instruction 



. to provide information services responsive to the needs of 
the individual user 



Origins of 
Information 
Mapping 
Features 



to make the preparation of learning and reference materials 
easier aid quicker than for conventional materials 

to make the task of maintaining and updating information banks 
both systematic and economical . 



Initially, research findings, generalizations, and procedures 
from many areas were considered with a view to their possible 
practical value for instruction or reference. 

Gradually we evolved the set of guidelines and rules for organ- 
izing and displaying information that we now refer to as Informr- 
t ion Mapping. These guidelines have their origins in such areas 
as these: 




. logical analyses of subject matters 
. learning research findings 
. teaching practice 
• programmed instruction techniques 
, display technology 
. human factors research 

. communications techniques , including effective writing 
principles. 

The implications of the various ideas were translated into practi- 
cal form and were documented as rules or procedures for preparing 
Information Maps, 

continued on next page 
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INFORMATION MAPPING: ITS BACKGROUND, continued 



Applications Inforination Mapping has bssn applisd to th© production of sslf- 

instructional books in the areas of probability , statistics, 
computer progranmiing, and Information Mapping techniques as well. 

Exploratory work with simulated computer displays and information 
block assemblies has provided some useful spadework on the kinds 
of options and display variations that a computerized system 
could offer to its users (Horn, Nlcol, Klelnman, Grace, 1969). 

Comment Because Information Mapping is of central importance an the 

present project. Chapter 2 will give a more detailed account of 
how came about and how it is done. 

However, it may be useful at this point to see the form of a 
map designed for self instruction. An example is shown on the 

next two pages: other examples appear in the appendix to this 
report . 
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EXAMPLE OF A CONCEPT MAP 



THE INTERSECTION OF TWO OR MORE EVENTS IS ALSO AN EVENT 



I nt roduct Ion 



De f ! n 1 1 i on 



Notation 



Diagram 



Comment 



Sorietimes we are Interested In an outcome that has more than 
one description - as, for Instance, In a car that Is yellow. 
Jaguar, and convertible. Such an outcome would be common to 
three different events. 



The I nte rsect I on of two or more events consists of those sample 
points that are common to the given events. These common 
sample' points In the Intersectfon or "overlap" of the events 
constitute a new event. The new event Is said to occur If an 
outcome common to the Intersecting events occurs . 



The symbol for Intersection Is n . PnQ Is read"the event 
'P Intersection Q," or "the event composed of the intersection 
of events P and Q." The event PnQ Is said to occur if both 
events P and Q occur, " 

The general case of the intersection o^ any number (^) of events 
can be wri tteni 

A, nA^n . . ,nA 
1 4 n 



The shaded part Is AnB 



S 

— B 



Outcomes In the "overlap" where events Intersect have more than 
one description since they belong to more than one event; for 
example, "both red“halred and blue-eyed*' is the intersection of 
two events; "left-handed, a pitcher, and a Yankee" Is the 
Intersection of three events; and "tall, dark and handsome but 
married" Is the Intfirsectlon of four events. 




continued on next page 



* From Introduction to Probability . Kleininan, Nicol, Grace, Horn, 1971. 
(In preparation.) 
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EXAMPLE OF A CONCEPT MAP* continued 



THE INTERSECTION OF TWO OR MORE EVENTS IS ALSO AN EVENT (continued) 



Example One 



Example Two 



Ex#mp 1 e Th ree 



A survey at a college asked each student about his Interest in 
these subjects* mathematics, psychology, government. Those 
with a strong Interest in all three were given another 
questionnaire about career plans. 

Event H - strong interest In mathematics 

Event P « strong interest In psychology 

Even.t G » strong interest In government 

Then the new event 
MnPnG 

represents those who received the 
second questionnaire. 



An urn contains ten balls numbered one through ten. One ball 
is drawn. 

Eveni % * the ball drawn has a number divisible by three. 
Event B a the ball drawn has a number divisible by four. 

AnB is the new event* "the b 
drawn has a number divisible 
three and by four." 

A - {3.6,9} 

B * {4,8} 

AnB ^ ^ 

Note. meiTiber of A are in B. AnB is the empty event. 

I f the 52 cards In a bridge pack are the 
definei 

Event H » all heart cards 
Event P ^ all heart picture cards 

The event HnP Is Identical with event 
P since all outCCTiies in P are also In H, 



1e space, we can 
shaded area * HnP 






Related 

Pages 



outcome, 14 
event, 23 



sample point, 17 
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MAJOR TASKS OF THE PROJECT 



Introduction A data base consisting of movable information segments could 

have the flexibility required for a system that aspired not only 
to serve various purposes but to serve them in a manner respon- 
sive to the individual inquirer. 

To accomplish the design of such a computer system, we outlined 
four major task areas and specified the end result we expected 
from each. 



1 Data Base 
Organi- 
zation 



2 The ■ 
Sequence 
Generator 



The rules and formatting policies for writing Information Maps 
were originally developed for making books for initial learning 
and reference use. Now to apply them not only to the organ! za— 
tion of computer data bases but also to the serving of other 
client classes means that they will need expansion and adapta- 
tion, For example j study of the reviewing and, briefing situation 
may show the need for additional maps to be made available in 
stoiage. Also it is clear that the data base must contain not 
only the information blocks themselves but also all sorts of data 
about the blocks Cname, content, difficulty level, etc.) so that 
the computer can judge their relevance for a particular user. 

The end product: The expansion of Information Mapping rules to 

cover the preparation- of information bases for computers and a 
document of detailed directions for carrying out the necessary 
procedures . ■ 



If we think of Information Mapping as a set of rules for parti- 
tioning the sentences about a subject matter so that they may 
be categorized, labelled and stored in the computer, we realize 
we need a second set of rules to govern the retrieval and 
arrangement of the information modules for display to the user. 



This second set of rules we have called the ^'sequence generator.” 
l^is will be the executive program that consults various kinds 
of data about the user and the state of the system before selec- 
ting from the information-block bank a series of displays appro- 
priate for the objectives and needs of that user. 

The design of such a sequence generator was another major task 
of this project. 



The end ^roduct - The specification of all classes of data re- 
quired by the sequence generator and a set of decision rules that 
indicate in detail the course of, the sequence generator's func- 
tioning in all user-system Interactions. 



continued on next page 
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MAJOR TASKS OF THE PROJECT, continued 



3 The The system must have a language through which the user can 

Command communicate his information needs and his reactions to the 
Language displays presented to him. 

The end product : A docimient defining each term in the set of 

commands designed for the system and specifying the effects such 
coimnands have on the system functions by way of the decision 
rules of the sequence generator. 



4 Documenta- The development of so complex a system is achieved through the 

tion- process of successive approximations. Tlius it can be predicted 

Updating at the outset that not only will there be a heavy demand for 

System documentation by the team of designers who are cfeatlng the 

system but also there will be an urgent need for updating pro= 
cedures that keep each team member in touch with the latest 
thinking of his colleagues. 



This fourth task of the project was to devise and follow systema- 
tic procedures for keeping track of the information developed 
in the course of the project. System documentation is to begin 
when the design project itself begins. 

The key idea is to test the utility of Infonnation Mapping as 
the fonn in which all system information will be written and 
stored. We will specify the detailed procedures for preparing 
system documents and for entering into multiple copies of them 
the latest available revisions. 

Procedures developed in the course of this task are expected 
to be applicable to the maintenance of the Leaming-Reference 
System in the operational stage -- that is, to chariging both 

course materials and system functions throughout the life of the 
system. In this aspect of the system, we expect the updating 
procedures to be automated and serviced through routine computer- 
directed Interviews with the updating clerk. 

The end product : A tested set of procedures for: 

. preparing system documentation In Infonnation Mapped foim 

. keeping system docimentatlon up-to-date during its evolution 

. maintaining the system at its highest efficiency durinfi 
real-time operations. 



continued on next page 
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MAJOR TASKS OF THE PROJECT, continued 



Project 

Document 



Availability 



Hie end product of the entire project : a document that gives 
specifications, procedures, decision rules, present limitations 
and possible future extenslor^ for various aspects of the system 

The objective was to develop the design in sufficient detail so 
that it is capable of being implemented by programniers who can 
adapt it for a given computer configuration. 

Project Document Qae presents the system design in a series of 
separate sections. Because of the extensiveness of the 
material, not all sections are equally available. The table 
below shows the main subdivisions of the documentation along 
with an availability symbol which is explained below^ 



SECTION 


TITLE OR DESCRIPTION 


APPROX. NO. 
OF PAGES 


AVAIL- 

ABILITY 


000- 

599 


Detailed plans for a S; quence 
Generator for use with Informa- 
tion Mapped Data Base^ 


ISO 


A 


600- 

699 


Detailed decision tables for 
implementing major functions of 
the Sequence Generator 


180 


A 


700- 

799 


A Reference Collection of Rules 
and Guidelines for Writing 
Information Mapped Materials 
Cdescribed in Chap, 2 of this 
final report) 


175 


A 5 B 


800 


A detailed description of the 
Information-Mapped Documenta- 
tion-Updating System (described 
in Chap, 6 of this final report) 


100 


A 


Index 


An index to section 600-699 


50 


A 



The availability symbols in the right hand coIuimi mean: 



A System docwnents are deposited with the 

Directorate of Systems Design and Development (MCDT) 
Deputy for Conmiand and Management Systems 
Hq. Electronic Systems Division 
L, G. Hanscom Field 
Bedford, Mass. 

B Available through 

Information Resources, Inc. 

1675 Massachusetts Ave. 

CMibridge, Mass. 02138 



continued on next page 
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MAJOR TASKS OF THE PROJECT, continued 



Related 

Doctiment 



COBOL 

Draft 



Information Mapping for Learning and Reference , by R. E. Horn, 
E. H. Nlcol, J. C. Kleimnan, andlM. G. Grace, ’ESD-TR=69-296. 
August, 1969, AD699201, (Available from Information Resources, 
Inc. 3 



As part of the project, we also wrote a brief segment of a new 
subject matter in order to extend the range of topics to which 
Information Mapping has been applied, TTie subject matter was 
COBOL and the draft of this is available as part of the system 
documentation under availability A above. 



IB 
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THE GENERAL IDEA OF THE SYSTEM 



Introduction The proje to desip a multipurpose system using Information 

Mapped data bases did not of course arise in a vacuum but in 
response to a felt need for more versatile training and refer- 
pce matpials in large computer facilities^ such as management 
infoimiation or military information systems. 

Such instructipal or training packages s are now available 
in these facilities are primarily sing -purpose programs serving 
all members of a given user class (such as initial learners) 
with almost identical information sequences. 

Hie capabilities of a large computer facility, whatever its 
primary purpose, are such that it could be a significant source 
of infonnation on all kinds of topics or subject-matter areas. 

Its capabilities are such also that it could be made to respond 
in a unique way to each individual in a variety of user classes. 



The 

Computer 

Environment 



It is against such a background that our initial thinking about 
system design took off. 



In the design we assinne the availability of a major multipurpose 
facility with certain characteristics which are described later 
However, because a high degree of flexibility has been built 

pto the design, certain versions can be implemented for smaller 
installations. 



Users of 
the System 



The Learning-Reference System then is planned to function either 
p part of a facility serving other pui^oses or alone as an 

infonnation service mainly for educational, training and refer- 
ence purposes . ® 

Hie system design is not tied to any given computer but can be 
implemented on a variety of currently available machines. 



The primary c 1 1 ent s we intend to serve are those who are directly 
concerned with using the subject-matter materials in some wayi 

. for initial learning 
. for reference work 
. for reviewing 
. for browsing 
. for briefing. 



continued on next, page 
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THE GENERAL IDEA OF THE SYSTEM, continued 



(continue d) 
Users of 
the System 



To serve these primary clients efficiently, the system must also 
be able to accommodate a secondary class of users who themselves 
serve the primary clients by operating the system, preparing 
new materials, gathering performance data, doing research -- in 
short 



, training supervisors 
. course designers, authors, editors 
. computer programmers 

. clerical aides 
, researchers 
. updating specialists. 



The As the design project got under way, then, certain aspects of 

Starting the system were established or assumed --we knew at least in 

Point a general way: 

. the kind of situation in which the system would operate 
. how the data base would be organized 
, the kinds of users it was to serve 
, some of the ways in which it should serve them. 



Comment 



From these outlines we began to work through to solutions on 
several Issues involving teaching strategies and the control 
of information sequenL,i.ng decisions. 
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SOME ISSUES RESOLVED 



Introduction Wl>en we consider how a sequence generator might be designed to 

serve the client who comes with a reference task, the problem 
seems fairly simple and straightforward == make the sequence 
generator fetch blocks on the topics selected by the client 
from a table-of- contents display or an index display. In short, 
make the generator follow the user's directions. 

But when we turn to the problem of how to serve the user who is 
learning something for the first time we encounter several 
troublesome interlocking issues of educational philosophy and 
teaching strategies. 

Since one of the most important services of the Learning-Reference 
System is to provide instruction for the initial learner, we 
must consider what kind of role the sequence generator is to 
take with the learner. 



While educational philosophies with their various goals and 
teaching methods differ in many ways, it is their diversity of 
opinion on the amount of control to impose on the student that 
concerns us here. ' " 

At one end of the spectrum, educators are convinced that the 
only justifiable goal of education is to enable the individual 
to discover and develop his talents and to grow in confidence 
in his oim capacity to find and learn whatever he wants. To 
produce independent learners, the role of the educational system 
is to provide the setting that encourages exploration and the 
exercise of individual choice and initiative with only a minimum 
of pressure from authorities or experts. 

If our system were to serve a similar goal, the student would take 
charge of his own learning and direct the generator to bring the 
sequences he chooses -- again, the generator would be designed 
to follow directions. 

Another variety of educational opinion holds rather similar views 
on the desirability of discovering and developing the talents 
of the individual but it goes about the task in a different way -- 
by testing and evaluating the student in order to plan a detailed 
program that the teacher believes will move him along toward 
greater development of his abilities. 

A sequence generator to serve this philosophy would need pro- 
visions for testing and for perfonnance recording, but most of 
all it would have to be able to prescribe sequences suitable for 
the student. The degree of control it would exert over the train 
i^fo^s-tion displays and test items shown to the student would 
presiMiably vary in some prescribed way with the abilities and 
progress of the student. 



continued on next page 
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Teaching 

Strategies 



SOME ISSUES RFSOLVED, continued 



(continued) 

Teaching 

Strategies 



Hie System 
Context 



At the other end of the educational spectrum are the fairly 
restrictive policies common to many conventional school systems 
("compulsory education") in which courses and their proper 
sequencing are prescribed by authorities with only minor 
concessions to Individual differences. 

The sequence generator to serve such a system would follow 
fairly simple rules of testing and then presenting pre-programmed 
sequences of information, digressing from the path only for 
remedial sequences when test items are failed. Such a sequence 
generator would be suited to handling programmed instruction 
as well. Compulsory response and response evaluation would be 
required to keep the displays coming. 

Many other in-between shades of opinion on the control issue 
exist but consideration of even these few points on the spectrian 
show that a sequence generator might be called upon to operate 
either under the client's direction or to assume full or partial 
control of sequence construction. 

While most of the educational viewpoints alluded to above are 
concerned with long-range goals of transmitting the skills and 
knowledge that will serve the student over a long period, it is 
probable that even the strategies for achieving short-term 
goals will show a divergence of opinion on how much guidance to 
give the student. 



In reaching a decision on the design of controls within the 
sequence generator, we also reviewed the context in which the 
system may be operating and the kinds of people who may be 
operating it and being served by it. 

Consider the environment in which the system may be used: 

. in a general education facility in which 

. both long-term and short-teimi goals are sought, and in which 
. the educators in charge may differ concerning how much 
guidance and direction should be imposed on students 

. in an industrial or military setting in which 

. training goals are paramount and the emphasis is on short- 
term mastery of specific information units, and 
. time constraints and other economic considerations may make 
system guidance of the individual the most feasible plan. 

continued on next page 



SOME ISSUES RESOLVED, continued 



(continued} 
The System 
Context 



Conclusion 



Conunent 



Regardless of the educational strategy that appeals to us per- 
sonally, we have to recognize that system clients come from the 
real world and have been exposed to various educational practices 
The clients may; 

* range from totally docile students dependent on guidance at 
every step to those who have acquired some degree of inde- 
pendence and initiative, or 

. be accustomed to setting up their own learning goals and to 
planning how to go about achievihg them. 



Build a system to provide; 

. complete freedom for the independent learner 
. varying levels of guidance for students who either ask for 
help themselves or are judged in need of it by the training 
supervisor. 

For the system to respond intelligently with any degree 
of guidance means that it must have ways of evaluating the 
infonnatic 1 needs and abilities of the student and that it then 
must know how to respond appropriately. 

The design task then is to fashion the system mechanisms that 
will accomplish the above functions. (1116 design task is^ not 

concem.ed directly with the infoimatlon contents of the system 

these can be as dull or exciting as the skills of the subject- 
matter writers permit.) 



While our discussion of controlling information sequencing has 
been focussed mainly on serving the initial learner, other 
clients of the system may need or prefer guidance as well. For 
instance, the person who comes for review may prefer a guided 
review provided by the system. Later chapters will describe how 
various degrees of guidance have been made available to other 
classes of clients. 
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DESCRIPTION OF THE TARGET SYSTEM 



Introduction 



General 

Features 



Once the control issue was settled, we examined other facets of 
the infomnation services a system might offer and gradually 
evolved a statement of the main features of the system. 

This description shov/s the general and specific functions we want 
the system to perfonn and the kinds of interactions it will have 
with various client groups. This is a more formal recapitulation 
of some points that have already been made. Some characteristics 
or functions are system-wide j others are specific to one class 
of users. 



piis system will work from a data base consisting' of categor- 
ized^ information blc.'’ks that are organized into maps (sets of 
blocks), units (sets of maps), and courses (sets'^f units) . 

From such a common pool, it will generate infonnation sequences 
that effectively meet the information needs of users with any of 
the following objectives ^ initial learning, reference work, review- 
ing, browsing, and briefing. Support personnel must be provided 
with the means for accomplishing their work as well. 

The means by which the system determines the Infoimatlon needs of 
its clients will be described presently. 

Hie function of calling up information blocks to be displayed to 
the user can be controlled: 

. totally by the sequence generator 
. totally by the user himself 

• jointly by the user and the generator with various levels of 
dominance by one or the other being specifiable . 

Clients without special training or previous ccmputer experience 
will be able to utilize the infonnation services of the system. 

TTie system will be interactive in the sense that the user can: 

, request and receive various kinds of Infoimatlon 
. select from menus and indexes 

. request m evaluation of his responses to practice questions 
. indicate a preferred order for the display of infoimation 
blocks from each map. 

Hie fact that these capabilities will exist in the system does 
not mean that they will be operative at all times- in fact, when 
the sequence generator has the dominant role in sequencing deci- 
sions, some of these functions may not be permitted to the user. 



continued on next page 



DESCRIPTION OK THE TARGET SYSTEM, continued 



Initial 

Learning 



Reviewing 



Reference 

Use 



Browsing 



Support 

Services 




The system will be capable of: 



^ Implementing different teaching strategies 

. assembling learning sequences to match each client’s 
background knowledge, aptitudes, and attitudes 

. prescribing differentially for those who differ in 

how much of a subject matter they want to cover and in how 
well they want to learn it 

. testing a student's knowledge of the field and its 
prerequisites 

. providing feedback to the learner 

. providing remedial sequences when requested or when 

a specified failure rate on feedback questions is reached. 

The system will be able to: 

. provide the user with a guided review tailored 
to his capabilities and objectives 

. supply support services to the independent reviewer 
who prefers to design his own review procedures. 

The system will be able to help the reference user find the 

information he wants without needing special training on the 

system. 



The system will enable the potential student or the casual brow- 
ser to explore the nature of any course in the system and to 
find out what might be required to learn it. 

The system will include special aids to simplify and systematize 
the work of those responsible for operating and maintaining the 
system. 
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COMPUTER REQUIREMENTS 



Introduci,ion Although w® have said that the Learning-Reference System has been 

designed without reference to a given coji^uter, it does never- 
theless require certain capabilities which are mentioned below. 

Although we envision the Learning-Reference System as part of a 
facility serving other purposes (management or command- and- 
control) , there is no reason v;hy a modified version of it could 
not function on a smaller scale serving only Instructional/train- 
ing purposes. 



Hardware 



In general terms the main requirements of the documented version 
of our system are theses 



. a third-generation digital computer or better 
. random-access secondary storage with read/write capability, 
including at least four tape drives 
. cathode ray tubes that are divisible into areas in order that 
text can remain on screen while other areas are changed 
. typewriter/teletype 



System software handles* 



. job-scheduling 
. multi-programming 
. time slicing 

interrupt analysis 
. input -output 
. random access 

Standard utility programs include: 



. card to tape 
, tape to r^dom access 
. sort— merge generator 
report generator 
. assembler 
. library maintenance 
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DESIGN STRATEGY 



Flexibility 



First Pass 



Summary of 
Strategy 



Ccmment 



In this first version, we did not expect the design of so com- 
plex a system to emerge full-blown from our drawing boards in 
untarnished perfection. Therefore we built into the system from 
the beginning the flexibility that would make it easy to change, 
add or delete aspects of the system. 

Flexibility was also required by the fact that at this stage the 
design was aimed at no particular computer. Wiatever computer 
system it might be implemented on would naturally entail some 
modification of the design. Especially might this be so if the 
main work of the facility were not educational and this design 
would have to be fitted into the other functions. 



Flexibility in the design would also be a boon during the devel- 
opmental testing period when the refinement of the system ^ould 
be guided by performance data. 



The ways in which flexibility has been provided for will be 
mentioned at various points as the system is described in fol- 
lowing chapters. 



Thus we intend in this first pass to sketch in with as much 
detail as possible those major structures of the system tiiat are 
not hardware-related and to specify the interactions among the 
various components that will accomplish the system objectives. 



At times, relativel)' simple solutions were adopted for the pres- 
ent version because it would be economically unwise to devise 
more elaborate mechanisms at this stage. For example, in the 
present version we assume that the information blocks are written 



at E given readability level. With little difficulty, however, 
the same information blocks could be written at, say, three 
different reading levels to serve a wider range of clients. The 
modification required to permit selection of blocks from three 
parallel banks instead of from one is simple, albeit tedious, but 
little is to be gained now from spending the time to work out the 
extra details in the absence of any finn need and in ignorance 
of how an unspecified computer could handle the extra storage 
space requirements. 



In short , then, we take as our task the description of how major 
functions can be accomplished, leaving elaborations or more com- 
plex alternatives until such time as a specific con^uter facility 
may have need of them. 



Certain of the design solutions 
explicit mention. They will be 



~H7 



accepted in this first pass merit 
discussed in a separate map. 



LIMITATIONS OF THE CURRENT VERSION 



Introduction 



Displays 



Computing 



Time Control 



Printout 
)i 



ERIC 



In Oider to separate the problem of creating a gene? T structure 
for the sequence generator from other more detailed ^.jblems, 

^ restricted the scope of this first version in several ways. 
These limitations and some of our rationale are discussed below. 

Throughout the pages of system documentation, it has been our 
practice to record notes concerning alternative solutions and 
possible directions for future extensions. 

The sequence generator does not keep track of or control the 
positioning, color, size, flashing, brightness or underlining 
of any text on the aisplay screen. We felt that selecting con- 
display was more important now than these other aspects 
which are dependent on system-specific characteristics. 

The design does assume that the amount of text displayed at one 
time will be adjustable by the client or by the system supervisor. 



In the future, we e^aect to have dynamic displays not only to 
Illustrate processes or trends but to guide reading and analysis. 

spiit-field scopes are avaiiable^ other opportunities for 
creative use of displays open up. In some installations such 
scopes will serve as a more convenient response medium since in 
addition to the main display, they can offer simultaneously lists 
Of topics or instructions from which the user can choose via 
lightpen action. 



■nie computing capability of the computer has not been integrated 
into the design of the Leamipg-Reference System. Again, it is 
of the specific computer that will determine how much 
f cm be put at the service of the user. Given 

sufficient capacity, the system can be amended to enable users to 
do on-line computations, problem solving, simulations and so on. 



l^e ability of the computer to keep track of the time duration 
of events in the system is useful in many ways, such as in keep- 
ing track of clients’ behavior. Identifying blocks that require 
an inordinate amount of time, and so on. There are teaching 
strategies also in which it may be desirable to put certain 
types of learners under time pressures * 



In this version of the system we 
lements . 



© no atten^t to formulate 



p would be of obvious utility to the major clients to be able to 
take home hpd copies of some materials. We expect this system 
to have such ppntout capability but its use and the nature of 
tne controls that would be necessary have not been defined. 



continued on next page 
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LIMITATIONS OF THE CURRENT VERSION, continued 



Single 

Track 



The 

Supervisor 



Teaching 

Systeni 

Operation 



Single 

Course 




The sequence generator’s present design operates on a single 
version of the information blocks in the data base. TTie blocks 
are written at a given level of reading difficulty. If the need 
arose to pi*ovide a wider range of levels, several versions of 
blocks could be prepared; either the supervisor or an internal 
decision rule could guide the generator to match the level with 
the individual user’s capabilities. The simpler solution would 
be to treat each version as a separate course; pretests would 
detennine which version was best suited to the client. Here 
the sequence generator would have the responsibility for 
assigning the user to courses. 



The role of supervisor has been created to perform certain 
functions concerned with getting the user established in the 
system initially and with determining how he will be treated 
by the sequence generator. 

Eventually all of these functions can be handled by the genera- 
tor. Through a computer-directed interview and testing program, 
the generator would gather the necessary user data and then 
refer to internal algorithms to find out how to interact with 
him in the session. 



At this time the sequence generator makes no coordinated effort 
to teach users about operating the system nor about Information- 
Mapping aspects of the information base. While we regard these 
as Important aspects of the final system, they are of relatively 
low priority at this stage of refinement. 

However, some provisions for explaining such things as the 
meting of certain commands have been included In this present 
version. Otherwise, we assume that instructional materials 
about the system are present in the data base and can be called 
up just as any other course materials can be. In the final 
system, the sequence generator will be able to Intrude hints 
about more effective use of the system into the interactions 
with the user. 



This version of the generator copes only with the progress of 
the learner through a single course, although it does provide 
mechanisms for switching courses and communicating about courses. 
Later we expect that the generator will be able to utilize data 
from other courses that the user has taken in the system and 
that it will be able to supply some interesting effects as a 
result of indexing across courses. 



continued on next page 
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LIMITATIONS OF THE CURRENT VERSION, continued 



Long-Terin 

Educational 

Objectives 



At this stage of system design, our services to the Initial 
learner do not include the teaching of those long-term 
educational goals that require prolonged exposure over courses 
to a teaching method or to a way of thinking. For example, we 
have in mind those long-range programs that are designed to 
promote the development of skills or of modes of thinking and 
problem solving. 

first priority was to build a sequence generator that can 
cope with courses and units of courses where specific learning 
outcomes have been outlined and reflected in unit tests and 
review tests. While these latter need not concern the inde- 
pendent learner who is pursuing his own course, they do afford 
the sequence generator with the means for prescribing for 
those students whom it is guiding. 
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CHAPTER 2 INFORMATION MAPPING 
OVERVIEW OF THIS CHAPTER 



Introduction 


Although Infomiation Mapping was introduced briefly in the 
previous chapter, we need to take a closer look at it since it 

provides the basic building blocks for this Learning-Reference 
System. 


This 

Chapter 


The topics that concern us in this chapter are i 

. the main features of Information Maps and their origins 

, previous subject "-mat ter applications and learning studies 

. general procedures for preparing Information Maps 

. special procedures that are needed to prepare maps for a 
computer-based system. 


Conunent 


Information Mapping is a growing, changing set of procedures 
and policies. It is probable that experience with a wider range 
of subject matters and with an operational computer system will 
reveal the need for new categories and new procedures. 

This chapter then describes the current state of Information 
Mapping. 



MAIN FEATURES OF INFORMATION MAPS 



Introduction 



Visible 

Features 



Invisible 

Features 



Infoimation Maps are conspicuous for their physical features 
the format in which they present information. ’ 

^ Information Maps, however, 

to rclasi^'o’T®"' is selected and organized ’according 

a classiiic-.tion scheme and a set of analytical procedures! 

The method of pr esentation and the organization of content 
may be thought of as the visible an Aisible features of a 



ITie more obvious visible characteristics are these: 

. information is presented in blocks 

. marginal labels identify the kind of i ‘formation 
in each block 

’ format is used for each kind of information; 

procedures follow one format 3 concept maps follow 
another distinct format, and so on 

. functional and uniform headings and subheadings are 
used to make scanning easy and to speed up reference 

. feedback questions and answers are located in close 
proximity to the relevant Information 



' provides map numbers 

for quick location of prerequisite topics. 

rseveral of these features are not used in tech nical reports.) 

arrangement and sequencing of materials presented in 
Infonnation Map fonnats are the result of: 

’ -ie«itf reference 



. specification of prerequisites for the subject-matter 



continued on next page 
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MAIN FEATURES OF INFORMATION MAPSj continued 



Ccontinued) 

Invisible 

Features 


• classification of the subject matter into , component 
types (concepts, procedures, etc.) 

. definition of the contingencies required for 
successful learning and reference. 


Footnote 


This map and the four following ones are slightlv modified 
versions of maps that appeared in the first account of 
Information Mapping by Horn, Nicol, Kleinman and Grace 1969 



INFORMATION MAP FEATURES DERIVED FROM LEARNING RESEARCH AND TEACHING PRACTICE 



Introduction Although Information Map features have their origins in 

several fields, there is no doubt that their principal 
foundations lie in education and learning research. On the 
chart below, we indicate briefly some of the findings that 
led to the design of certain Information Map features. 

This chart (which is not intended to be exhaustive) is one 
example of the research support behind Information Mapping. 

Naturally, the evidence is not all of equal strength, but 
we have tried to bring to bear on a practical task some of 
the most promising factors. 

Because the experimental basis for some map features is 
extensive, we cite wherever possible research review articles 
to put the reader in touch with the main sources of evidence. 
In the citations below, such major review articles are 
marked by asterisks to distinguish them from reports of 
original research . 



These results of sducational 
research lead to . . . 



Active responding generally 
aids learning. 

(Lumsdalne and May* *, 1965 • 
Briggs*, 1968| Glaser*, 1965) 

The act of writing responses 
helps some learners. 

(Edling*, 1968) 

Feedback or knowledge of results 
(or 'reinforcement*) often 
facilitates learning by' 

confirming or correcting 
learner's widerstandlng 

* providing a motivational 
effect 

• improving scanning, be- 
havior 

(Lumsdaine aid May*, 1965 j 
Smith , 1964] Gagni and Rohwer*, 
1969] Glaser*, 1965) 




.. . . these in^lications for the 
design of instructional materials. 



Insert feedback questions after 
introducing new materials , 

Locate answers co.nveniently nearty. 



continued on next page 
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INFORMATION MAP FEATURES DERIVED FROM LEARNING RESEARCH AND 
TEACHING PRACTICE, continued 



These results of educational 
research lead to . . . 




. . . these in^li cations for the 
design of instructional materials. 



The insertion of questions, 
"test- like events," after text 
segments has a positive effect 
on learning. Giving Knowledge 
of results further increases 
the effect. 

(Gagne and Rohwer*. 1969 ■ 
McKeachle*, 1963) 


L> 

i 


Use feedback questions after maps 
with new information and use sets 
of review questions after natural 
1 clusters of maps and at the end 
j of topic treatments. Provide 
1 answers as well . 


Self-tests, pretests facilitate 
retention, 

(Glaser*. 1965* Briggs*. 1968; 
Bloom*. 1963) 







In concept learning . a variety 
of examples promotes learning. 
(Gagni and Rohwer*. 1969 • 
Lumsdaine*. 1963) 






Use examples and nonexamples to 
point up differences and similar- 
ities among concepts. 








Instructions are useful in 
calling learner's attention to 
important features . 

(Gagni and Rohwer*, 1969; 




> 


Use introductoity paragraphs 
or previews to alert learner to 
importance of upcoming ideas. 



Gagnfe. .1965) 



Judicious use of underlining 




Underline important words in 


often helps to focus attention 






definitions . 


on key elements. 








(Hershberger and terry, 1963) 
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INFORMATION MAP FEATURES DERIVED FROM 
TEACHING PRACTICE, continued 



LEARNING RESEARCH AND 



These results of educational 
research lead to . . .. 




► 


. . . these ii^llcations for the 
design of instructional materials. 








"Cueing” or labelling appears 
to aid by alerting learner to 
nature of upcoming infonnation 
and informing him what his 
leaming task is, 

(Glaser*, 1965) 




> 


Us a marginal labels and 
Informative map titles , 








Pictorial materials often 
help leaming, 

(Briggs*, 1968) 




> 


Use diagrams and drawings to 
illustrate concepts and procedures. 



For some kinds of materials , 
charts of the infonnation are 
valuable. 

(Feldman, 1965) 




Use tables amd verbal 
to display concept 



Simple sentence structures in 
the active voice make leaming 
easier. 

CGagn§ and Rohwer*, 1969] 
Coleman, 1965) 




In general, use active 
and single sentences. 



voice 
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ORGANIZATION AND INTEGRATION OF INFORMATION IMPORTANT FOR LEARNING 



Introduction 



Theoretical 
Dis CLUBS ions 



Implications 

for 

Information 
Map Books 



Some important features of information mapping owe their 
origins to a topic of current theoretical interest among 
learning psychologists - namely, the logical and psychological 
structures of knowledge and their impact on learning and 
retention. 



Piaget had long ago speculated that ’'learning ... is 
facilitated by presenting materials in a fashion amenable 
to organization" (Flavell, 1963), but it is only in recent 
years that psychologists have actively taken up the problems 
of how cognitive structures develop and of the role of organ- 
ization in leaiming and retention. 

TTie 'atomistic' approach of most prograiraned instruction ma- 
terials has been criticized CStafford and Combs, 1967) and a 
firm case made for the advantages of "meaningful organization 
and holistic presentation of materials." 

In a syn^osium on "Education and the Structure of Knowledge" 
(Phi Delta Kappa, 1964), P.H, Phenix remarked: "It is 

difficult to imagine how any effective learning could take 
place without regard for the inherent patterns of what is 
to be learned. " 

David Ausubel (1960, 1963, 1964, 1968) has developed a logical 
and psychological case for believing that learning aid long- 
teimi retention are facilitated by 'organizers' which provide 
an 'ideational scaffolding.' He has now amassed considerable 
experimental support for his hypotheses. 

The well-known studies of Katona (1940) with college students 
pointed iq) the importance of organization for learning and 
for retention. 

The relation of organization of materials to ease of learning 
also finds support in the area of verbal learning research 
(Underwood*, 1966). 



Although many issues remain to be settled by research, a 
strong case can be SL^ported both logically and empirically 
for the advantages of organizing and integrating features in 
materials for leaming. Both verbal md graphical means can 
be used to inject a sense of organization and direction into 
a subject-matter presentation. 



In the practical effort to design effective leaming materials, 
we have incorporated a number of features intended to help the 
learner integrate and organize the ideas for more efficient 
storage in memory, TTiese are listed on the next page. 
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FEATURES TO AID IN ORGANIZATION AND INTEGRATION 



Introduction The following list of features designed to promote integra- 
tion of concepts and relationships contains some that we 
have already adopted on other groLinds, For instance, the 
guidelines called for practice questions and answers through- 
out the text because learning research suggested their 
value in several ways; but questions can also be phrased 
to encourage integration of ideas over sections of learning 
materials . 

Examples of maps showing some of these features are given in 
the appendix. 



List of Features that aid in the organization of ideas are: 

Features 

reviews and previews: to take stock of the 
ideas developed up to that point and to prepare 
the ground for relating them to new concepts 
about to be encountered 

. introductions to each map: to relate new idea to 
previous concepts or to familiarize with nature 
and importance of new idea 

. recaps or capsules: to summarize succinctly 
the essential ideas of rules or principles 
in nutshell form 

. tree diagrams: to sketch the ideas and pro- 
cedures of a topic so as to show the role of 
each and its links to others 

compare-and-contrast tables: to point up the 
similarities and differences between two concepts 
that are sometimes confused 

. summary tables: to chart in easy reference 
form the main concepts of an area 

. review tests after short sets of maps and at the 
end of units : to promote the integration of 
several concepts and to practice using them in 
problem solving 

. prerequisite charts: to show schematically the 
paths the learner can take through a subject 
matter in order to reach the learning objectives. 
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INFORMATION MAP FEATURES FOR EASE OF REFERENCE AND REVIEW 



Introduction In designing materials for initial learnings we added features 

fs-cilitate the return to ideas previously encountered, an 
activity that is often frustrating with conventional materials 
where the contents of the paragraphs are mostly unlabelled. 

ommon sense ^ human factors research^ and graphic technology 
were used in formulating aids for easy retrieval of information, 
A list of these aids appears below. 

clear also that these same feat’ " would be important 
in Infoimation Mapped b^^^ such as reiti’ence manuals or Job 
aids. In the computer-based Learning-Reference System, these 
same features are still present, although they are of less 
importance because the sequence generator can take on the tasks 

quickly locating information needed by referencers and 
reviewers . 



ist of Features that aid in locating information are: 

Features 

. Tables of Contents are organized and formatted 
to speed location of topics and special features. 
CThis report does not use the standard fomiat 
but follows certain BSD report requirements.) 

. A predictable format for each type of map 

(concept, procedure, etc.) facilitates location of 
needed information. 



. Map headings in consistent typography help in 
scanning for map topic. 

. Marginal labels help not only In locating the kind 
of information sought but also in skipping that 
not required. 

. "Related Pages" or local indexes at end of each map 

permit quick location of concepts prerequisite to the 
given map , 

. Decision tables display the choices appropriate 
for each possible situation. 

• Sumniary uables assemble main facts and relations 
for easy review and reference, 

, Capsules provide "kernel" statements of key rules 
or concepts , 

, Diagrams show graphically the sequences of events 
in a process or procedure. 
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. Indexes aid information retrieval. 
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EXPERIENCE WITH INFORM<^TION MAPPED MATERIALS 



Introduction Our experience with Information Mapped materials can bt 

discussed under two headings : 

. the range of subject matters to which the techniques have b 



een 



Subject- 

Matter 

Experience 



the learning results 
search studies. 



the use of these materials in re- 



Subject-matter experience is relevant to the Information Map 
classification system because this system evolved from attempts 
to categorize the kinds of topics dealt with in certain subject 
areas. Undoubtedly the kinds of information we have tried to 
handle will color the category scheme and uridoubtedly we shall 

have to add terms when we confront entirely different subiect- 
matter areas. - 



er|c 



ior part of our experience has been ga: ned in writing 
testing, and revising a ISO-page introduction to sets and proba- 
bility. This book of mapped leaniing materials constitutes a 
ten-hour self-instructional course. It served as the basis for 
several research studies mentioned below. Recently several 
mop units of approximately 100 pages have been added to it. 

Other volumes in the statistical field are currently in prepar- 
ation, - ^ 

entpe dopmentation for the Learning-Reference System has 
ep wripen in Information Map form, as the previous chapter 
noted. It consists of several hundred pages. 

^e stpf training/reference manual for Information Resources 
Inc., IS Information Mapped and so arranged that short courses 
CM be assembled from it to suit the needs of various classes 
of new personnel, such as temporary typists, map writers, 
editop, updating clerks, and so on. Iliese self-instructional 
materials may take as little as one hour for temporaiy typists 
or as much as several weeks for map writers. 

Other subject areas with which the system has been tried are: 

APPROX. NO. 



computer prograimning 
the binary number system 
CONVRS, an experimental computer language 
CANARD, a simulation iMguage 
ptroductign to descriptive statistics 
introduction to matrix algebra 
peiTnutations , combinations, and the 
binomial theorem 
introduction to COBOL (see p. 12 for availahjlity) 25 



OF INFOMAPS 
75 
60 
ISO 
ISO 
75 



50 
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EXPERIENCE WITH INFORMATION MAPPED MATERIALS, continued 



T^tinued) 
^ ect- 
Matter 
Experience 



Learning 

Results 



In addition, brief units have been written for: 

. basic concepts of operant conditioning 
. some topics in American history 

, two dentistry topics: how to extract a tooth, and 
peri odontology 

. a topic in chemistry: the structure of the atom 
, the Munsel color system in art 
, darkroom procedure in photography. 

The Information Mapped book on sets and probability served as 
the research vehicle for several evaluative studies of initial 
learning by college students. The results (reported in Horn, 
Nicol, Kleinman, Grace, 19693 showed significant achievement 
scores and the students rated the materials attractive and 
valuable. 



Although we have made no other formal studies of learning from 
Information Map books, the short training courses for staff 
personnel have been of considerable practical importance. 
Because the courses are self-instructional with self-tests and 
practice exercises, they enable us to obtain productive work 
from even temporary typists with only a minimum investment of 
staff time in training them. 
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MAP CLASSIFICATION RULES AND WRITING GUIDELINES 



Introduction Thus far we have talked about what Information Maps look like^ 

how they came to look that way, what kinds of subject matters 
have been mapped and how maps have been used. 

Now, in preparation for extending the ideas to computer data 
base organization, we must take up the topic of how Infomation 
Maps actually get made- -how the content of a subject area gets 
transformed into labelled information blocks. 



Planning Most writing of information banks about a subject matter will 

probably start with writing for the beginning student who 
knows little or nothing about the area. Curriculum planning 
can proceed according to whatever educational philosophy the 
author prefers . 



We suggest that eventually the outline of topics to be covered 
be translated into a prerequisite chart Csee example, appendix), 
a schematic display that works backward from program objectives 
to the topics required to meet those objectives. Such a chart 
depicts in network format the relation of each concept or pro- 
cedure to others and naturally suggests some sequencing possi- 
bilities. 



The 

Mapping 

Guidelines 



From this chart the author works, following the extensive set 
of guidelines for writing, formatting, and classifying informa- 
tion, These guidelines also assist the author in identifying 
the materials that have to be written to serve reviewing, ref- 
erence and browsing purposes as well. 



An important part of the Information Mapping documentation is 
the set of guidelines for classifying information and for writing 
the information blocks appropriate for the different types of 
maps. The guidelines themselves occupy several hundred pages 
(Infonnation Mapped, of course) of format policies, instructions, 
and illustrations (reference given. Chapter 1 on page 11) . 

Tabler showing the present ciassifi cation of map types and per- 
missible infojnnation blocks are given in the appendix to this 
report. These lists are not considered complete nor fixed for 
we regard Information Mapping as an evolving system. Wider 
experience with varied subject matters will show other categories 
that are needed to cover the information comfortably. 
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MAP CLASSIFICATION RULES AND WRITING GUIDELINES, continued 



(continued) 

The 

Mapping 

Guidelines 



The guidelines originated with the need to specify rules and 
policies for writing book-type materials. In the course of the 
present project they have been extended to cover the preparation 
of information banks for the computerized system, 

types of supplementary maps, that is. maps containing 
no new information . are required for initial learning and for 
reviewing, browsing and reference needs as well. Some of these 
maps are : 



, previews 
. reviews 

. corapare-and- contrast maps 
, summaries 
. decision charts 



Comment 



Develop- 

mental 

Testing 



Tables of contents and Indexes are also among the materials that 
have to be prepared. For all of these, writing rules, formatting 
policies and when-to-use guidelines have been documented. 



iVhlle the guidelines suggest some principles that often make 
for effective writing, it is the author who must assume the 
ultimate responsibility for the style of writing in the blocks 
of his course. The course materials will probably be as varied 
in content and approach as are individual tutors- -imaginative, 

boring, humorous, pompous, terse, verbose, challenging, trivial 
and so on. 



No matter how inspired the formulation of guidelines may be, they 
can never eliminate the need for developmental testing. For all 

Information Mapped materials empirical testing and revision are 
standard policdes. 
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MAPPING PROCEDURES FOR COMPUTER APPLICATIONS 



Introduction 



Documents 



Modified 

Procedures 



^en Infomation Mapping is applied to the preparation ana organ- 
ization of computer information bases, certain new policies or 
procedures are required. Some of these are only slight variations 
of the rules for writing books but others entail extra writing 
tasks for the authors . 



As the design for the Learning- Reference System evolved, the 
documentation of new writing requirements kept pace. This 
is kept up to date in the project document Ccited in Chapter 
1) and is available for the immediate training of course writers 
for the new system. 

It is possible that in actual practice some of the map prepara- 
tion tasks will be assumed by mapping specialists other than 
authors. For the present, we are concerned primarily with iden- 
tifying the general nature of the procedures now needed in a 
computer application and not with allotting functions to personnel. 



TTie guidelines for planning, writing and formatting Infonnation 
Maps remain essentially the same. In writing for the computer 
however, there is the additional requirement that 
blocks be independent in phrasing; since these blocks will even- 
tually be assembled in unspecified configurations, we kist 
avoid such phrases as "as we mentioned in the block above'* or 
the previous map explained..,," and the like. 

A major modification for the author is that he must prepare much 
more material for the computer base. Whereas in writing for a 
book he may find through experience that three or four worked 
examples are sufficient, with the computer system he will be 
dealing with a wider range of user abilities and thus needs a 
larger store of worked examples and practice exercises or 
feedback questions. 

An earlier map mentioned that the author writing for the Learning- 
Reference System must also provide special maps for users with 
browsing, briefing, reference or review needs, Hiis means that 
he will need to write Information blocks about the course itself, 
including such materials as how much time it usually takes, what 
its prerequisites, what are the main ideas and major appli- 
cations (if ay), and so on. 



continued on next page 
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MAPPIiNG PROCEDURES FOR COMPUTER APPLICATIONS, continued 



New 

Writing 

Requiremen 



User 

Character- 

istics 



Hie main effect of the computer’s advent into the picture is to 
increase the number of writing tasks. Tliese may be categorized 
:s under two headings : 

. telling the computer about various classes of potential users 
of the subject-matter materials, and 

. telling the computer about various aspects of the information 
materials themselves. 

It is up to the author or subject-matter expert to record for 
the sequence generator any rules or recommendations he may have 
for handling users with different background characteristics. 

For example, the author of mathematical materials might recom- 
mend that initial learners with an antipathy for mathematics 
should be shown certain types of blocks that would be boring 
for learners who had a strong background in mathematics. Such 
advice from authors will be mainly of importance in dealing 
with initial learners. 

Among the classes of Information that an author might recommend 
as bearing on information sequencing decisions are these: 

. intelligence quotient 
. vocabulary level 
. areas of major interest 
. subject-matter compete: ce 
. attitude toward subject area 
. job and socioeconomic status 
. pretest scores 

In each case in which the author formulates a relationship be- 
tween US', r background data and sequencing treatment, it is hi- 
responsibility to specify as well how the user data are to be 
obtained. Some may be collected on-line using interview ques- 
tionnaires or tests provided by the author or they may come 
from specified off-line sources and be entered into the computer 
by supervising personnel. 

Pretesting will provide a major source of guidance to the sequence 
generator since it will have an influence on the selection of 
maps to display to the learner. 
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MAPPING PROCEDURES FOR COMPUTER APPLICATIONS , continued 



Course 

Character- 

istics 



Cominent 



o 




Many characteristics of the course and its various parts have to 
be identified and stored in the computer in a fixed format in order 
for the sequence generator to carry out its task of assembling 
information block: Some of these characteristics must be indicated 

by the author while others can be supplied by support personnel. 

Some examples of the kind of information the authors must provide 
are: 

, difficulty rating for each feedback question 
. instructions for judging all possible user responses 
, instructions for remedial sequences following each incorrect 
response 

. possible following maps. 

Numerous characteristics of each course, mau and information block 
must also be coded for the sequence generatcr but these can be 
determined by support personnel following routine checklist pro- 
cedures. 

More about the special aids to authors appears in Cliapter 5. 

The vrtrious characteristics of users and courses will be 
encouvcered again in the next chapter where data base 
organization is described. 
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CHAPTER 3 THE CONCEPT OF A SEQUENCE GENERATOR 
OVERyiEW OF THIS CHAPTER 



Introduction The heart o£ the Learning-Reference System is the executive pro 

grajn that acts as intermediary betvveen the user and the stored 
iori blocks. It can either fetch the information re- 
quested by che user or it can recommend suitable sequences for 
those who want or need guidance. 



This 

Chapter 



The Next 
Chapter 



These functions are accomplished by a set of rules and procedures 
to which we have given the name ’*the sequence generator. ' 

To respond to the different purposes of different kinds of users 
wanting to interact with different types of subject matters, 
the sequence generator must be capable of tapping a wide range 
of information if it is to deliver sequences of suitable di 



The kinds of information the generator needs and the mechanisms 
that enable it to -work are sketched out in this chapter. The 
command language comes in here also as the medium through which 
the user interacts with the generator* 



In the next chapter we will shov^ how the mechanisms enable the 
sequence generator actually to respond individually to the 
person who comes with a learning purpos^ in „iind. How the 

system psponds to other classes of users will be described in 
tne following chapter. 
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A SEQUENCE GENERATOR: THE GENERAL IDEA 



Introduction If a common data base is to serve multiple purposes, there must 

be a guidance system for selecting, assembling and displaying 
those information blocks that are appropriate for each user’s 
purpose and for his own unique pattern of capabilities, atti- 
tudes and personal preferences. 



Definition The sequence generator is the set of rules and procedures for 

selecting lind arranging information blocks into a sequence of 
displays appropri ’te for a given user. 

Nature of The sequence generator is written as a series of decision tables 

the Sequence that check on the values of r +ain stored data and then take 

Generator the actions that are prescribea for the value configuration found. 



The stored data are of many kinds : 

. ab'Ut the user, his abilities, preferences, and so forth 
. about the system itself, how to use it, what kinds of materials 
it contains 

. about each course and its various subdivisions even down to 
the individual information block and practice exercise 
. about the state of the system at the moment as shown by various 
counters and indicators. 

As the user interacts with the information base, the sequence 
generator keeps track of the action by updating the indicators 
and by entering new data into the system. 

Example Since the decision table is a key ingredient in the system, we 

illustrate it first in a very simple form. At the end of this 
chapter a more complex example will be given. 

Let us imagine that the Learning —Reference System has been 
activated for the user aid that the computer has asked the 
person to Identify himself by typing his name and code number. 

Tile computer must then examine rhe response to see wh^tner the 
person is authorized to use the system. The way it does this 
is shown very simply in the following decision table which we 
arbitrarily entitle TABLE 100 WHO IS IT. The table is ej^lained 
below. 
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A SEQUENCE GENERATOR: THE GENERAL IDEA, continued 



(continued) 

Example 



Explanation 
of Example 
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yt 


\BIE 100 WHO IS IT 


1 


2 


3 


c 

0 

N 

D 

I 

T 

I 

0 

N 

S 


Is his identification legal? 

(Does his response match any item on 
LEGAL CODE LIST?) 


Y 


N 


N 


Has he given an illegal identification 
once? 

(What is the value of TIME FLAG?) 




N 


Y 

1 












A 

C 

T 

I 

0 

N 

S 


Display: ''Illegal identification" 




X 


X 


Put Y (yes) into TIME FLAG 

(Shows one illegal identification given) 




X 




Display: "Identify yourself" 




X 




Put user's nMie into CURRENT USER STRING 


X 






GOTO TAB*.B 100 WHO IS IT 




X 






GOTO TABLE 105 rHICH MODE 


X 








Stop interaction 






X 





Hie top section of a decision table records the answers to 
questions about some current condition. 

Hie lower section of a decision table shows fhat actions are to 
be taken for each relevant pattern of answers. — 

l^e table is read by considering the pattern of answers given in 
the first numbered column and then looking down the column to 
see what actions are prescribed (shown by X's). And so on, for 
each column. 

In our exan^le, when the answer to the first question is Yes, 
then we have no interest in the answer to the second question 
about whether an unacceptable answer was given before- -that is 

why the answer box for the second question is left empty in 
Column 1 , 

continued on next page 
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A SEQUENCE 

(continued) 
Explanation 
of Example 



Comment 



GENERATOR: THE GENERAL IDEA* continued 



If identification i '■ legal j the computer takes the actions 
marked by X's in the lower half of Column 1, namely: the user’s 
name is put into a computer storage location called ''Current 
User String" and the computer goes to Table 105 WHICH MODE to 
Tnd out whether the user's purpose is learning, browsing 
reviewing, etc. 



Column 2 in the table shows what the computer does if the user 
has given an illegal response to the first query--"NN" is the 

pattern of answers to the two questions. We see that the com- 
puter: 

tells the user the problem 

. records in a place called TIME FLAG the fact that one 
illegal answer has been given 
• again asks the user to identify himself, and 
, then returns to step through this same table again. 

If the user gives a proper code this time, we are of course 
back in Column 1- but if a false code is given a second time 
we have the situation in Coliunn 3: an illegal response has 
been given and the value of Tim FLAG tells the computer that 
this has happened once before. Ihe computer then informs the 
user that his response is not legal and refuses to continue 
the interaction. (An accredited user who was making some 
error would presumably consult the supervisor at this point.) 



very simple 
important points: 



e of a decisi 



on table illustrates two 



the way that certain indicators within the computer get 
changed to keep track of events in interacting with the 
user--in this example, storage locations called "flags" 
"strings" were modified 



the interlocking nature of the set of decision tables 
we call the sequence generator- -the example shows 
the flow of action may go back through the same table 
again or may go on to other tables. 



Although the actual decision tables of the Learning-Reference 
System are vei^r much more complicated, even this simple examp] 
may give some feel for the way that a mere set of tables can 
constitute a sequence generator. 
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A SEQUENCE GENERATOR; THE GENERAL IDEA, continued 



(continued) 

Comment 



The decision table offers tlie important advantage of flexibility 
which was discussed in the first chapter. An evolving system 
like this one needs components that may be modified easily 
without causing expensive changes in other structures or even 
in system documentation. It is easy to see that changing a 
line or column in a decision table would be simpler than changing 
a flow chart. Explicit procedures enabling support personnel 
to keep iecision tables up to date are part of the aids supplied 
by the Documentation-Updating System described in a later chapter 

The use of decision tables simplifies the programmer’s work 
since a nunher of compilers accept key-punched decision 
tables. 



Later in this chapter we shall present a decision table of 
more complexity, but first we describe those other aspects of 
the system that provide the data for the decision tables. 



SCHEMATIC OF THE INFORMATION 



SEQUENCE GENERATOR 



PRIMARY 
MODES OF 
INTERACTION 

. Initial 
Learner 

. Referenc 

. Reviewe 




USER 

. Training 
Superviso: 



. Author 

, Computer 
Programme 

. Cle-ical 




TERMINAL 

SCOPE 

AND 

KEYBOARD 



SEQUENCE GENERATOR 
PROGRAM 



responds to user 
coimnands and needs 

sets goal and 
criteria 

computes decisions 
about next 
displays 

administers tests ^ 
evaluates results, 
and gives feed- 
back to user 

provides routine 
updating of 
various data 
bases 




TTiO Sequence 
Generator 
finds out in 
which of these 
modes the user 
regards himself 
and provides 
appropriate 
services . 



Designed to operate not 
only under total user 
control and total system 
control . but also at any 
one of numerous points 
in between. 






DATA BASES USED BY 
SEQUENCE GENERATOR 



Courses, composed 
of 

. Inf(^-*"mation Maps 
divjded into 
. Information 
Blocks 

. Questions: 

. pretest 
. feedback 
. review 

User Data Bases 
for each user 
composed of 

. user back- 
ground data 

. current status 

. place-keeping 
data 
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MODES OF OPERATION 



Introduction 



Description 



MODE 

Initial 

Learner 



One of the primary objectives of the Learning-Reference System 
is to serve equally well users who come to the system for quite 
different reasons . ' 

Within the sequence generator certain sets of decision tables and 
other mechanisms have been designed to serve each of the separate 
pui^oses for which users might come. — — 



A mpde of operation i s a set of rules for sc -ving the needs of 
^ pa-rticular type of user. It includes decision tables, sw 



settings and data storage locations especially designed to 
assemble the kind of displays needed by a given class of user. 



the most c 



Because the needs of user groups frequently overlap, the modes 
share many elements and computer routines. The mode requiring 
Dlex mechanisms is that for initial learning; hence 
le was designed first so that later r les could make use 
of its components. 

In the current version of the sequence generator thirteen dif- 
ferent modes have been tentatively defined and these are described 
briefly below. The modes for most primary clients of the system 
nave been worked out in detail but some of the support modes 
have been defined only in general terms and may eventually be 
consolidated into two or three modes. 

A mode is brought into operation by setting a ’’mode switch” to 
the sequence generator what kind of person is using the 
system. (At present such a switch setting would be made by a 
supervisor in consultation with the client, but this task will 
eventually be done automatically through a user-system dialogue.) 



DESCRIPTION 



The pui^ose of this mode is to provide sequences of materials 
for those who come to learn something for the first time or to 
re-learn something mostly forgotten, 

A major goal is to deliver leaming materials that are chosen 
especially to match the needs and interests of the individual. 

Whether the learner chooses his own path through the material or 
IS guided to a greater or lesser degree by the tutorial provisions 
of the sequence generator depends upon a decision that is made 
by the supervisor or by the learner himself. 
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MODES OF OPERATION, continued 



MODE 


DESCRIPTION 


Referencer 


^fTie function of this mode is to enable a user to retrieve infor- 
mation from the data base easily and quickly, 

ine user controls his interaction with the system and receives 
relatively little guidance from the generator except for some 

prompting when inappropriate conmiands are used or when he asks 
for help* 

The reference user needs little knowledge of the systeni since he 
can acconiplish most tasks quickly with only four co^ nds* 


Reviewer 


This mode serves those who wish to revievv a subject area they 

were once faniliar with to some unspecified degree. It provides 
a user with : ' 

. a quick assessment of his knowledge of the subject area 

. an individualized review of those aspects that he has for- 
gotten or wishes to see again 

‘ re-learning sequences where necessary either under system 
control or under the user’s control. 

The aspects of the system that are to be controlled by the user 
or by the system are assigned by the sunervisor or reviewer. 


Briefing 


Intended primarily for the potential user of another mode, this 
mode enables the individual to find out about the general nature 
and content of any subject matter stored in the system. 

Hie user of the briefing mode can see many special maps describing 
a given course, its difficulty level, an estimate of time re- 
quired to learn it, some of the big ideas of the subject, its 
applications and so on. 

The degree of sequence control allocated to the user and to the 
system can be specified for this node also. 


Browser 


Browsers need the tools to explore the information base in any 
mnner they like. This means they need access to all courses, 
^e system provides aids in terms of suggested commands and 
echniques that minimize the need for special trainino- on the 
system. This mode has not been defined in detail. 
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MODES OF OPERATION, continued 



MODE 


DESCRIPTION 


Supervisor 


fusef fatfbf.f"' information abouT* 

a user that the sequence generator needs in order to serve him. 




usLfanffff required about the primary 
obtained^frol “®®.*^''® s/stem. The information is 

hThf liff e neL f f ® ®*™®‘«red interview. Thus 

over fsLr A ^ language and little control 




expect the supervisor functions to be accom- 
plished by internal algorithms and not by support personnel 


Explan atory 
Note 


«Lfbirf described in general ^ 

forTfofthfs^fe^fof^f^^^tvfef^^"*^^ decision-table 


MODE 


DESCRIPTION — 1 



Instructional 
Designer 



Author 



or 



These users have the responsibility for specifyine the seouen. 

ato? w'’0 are guided by the sequence gener- 

® P’^°''ides such on-line aids as direcLf 

nterviews for eliciting the required information from them. 



Authors create course materials. They work in Fn^-Ush and need 

may occSona?^ aspects of formatting. They 

they rareirin^Lact S®"®ral 

- ^ inreract ivith the sequence generator. 



Clerk 



inserf Selftf f demand and to 

^ ^ ' or replace materials. TTiey hare access to 

usage summaries and to clients' comments on the material! 



Progranuner 



Cleracal workers who prepare course materials for .,ie comriuter 

nfeL:t"f "Tw'"- Provides aids un- 

operating Se systef 



^fwif i“Pl®“«"ting the system design 

^ thorough^nder. 
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^lO^ES OF OPERATION 3 continued 



MODE 


DESCRIPTION 


Researcher 


These users need suiranaries of client-system interactions and 
various tables of data. They must have access to the user data 
bases. The system provides the support programs that enable 
them to extract the information they need in the form they 
prefer. - 


Proctor 


Some versions of the system may operate with an attendant for 
the area in which user stations are located. Such personnel 
should be able to answer user questions, suggest courses of 
action, explain ambiguous situations and the like. They may 
need access to the user-system data in order to answer some 
questions about the interaction. 



DOMINANCE 

Introducti 



Control 

Switches 



?0LS 



Two pieces of information have a marked influence on the way the 
sequence generator treats the client--first , his purpose in 
coming as indicated by the mode he is going to operate in, and 
second, the degree of control the client is permitted to exercise 
over sequencing decisions. 

Tlie system is set up, as Chapter 1 explained, to operate at any 
point on the continuum that extends from total user control to 
total system control of the information displays . 

Now we describe the m.echanisms that tell the sequence generator 
just where on this continuum the current client stands and what 
aspects of the system (if any) he is to control. 



The amount of control that the user is able to exercise over the 
system is defined by a series of "switches." Here a "switch" is 
not a physical knob- like or lever-like gadget, but it acts like 
one, it is actually a variable, a quantity stored in commiter 
memory, that can take on different values just as a switch 
setting can. For instance, the Mode Switch tells the sequence 
generator what the user came for and it is "set" by inserting a 
number between 1 and 13 in the computer location called Mode 

Switch. The ntunber 1 shows that the client is an Initial learner, 
and so on. 

Some other dimensions that are controlled by switches are such 
things as whether the client can use certain commands (such as 
to go forward or go back) , whether he should see and/or pass 
feedback questions, whether he can change the order of block 
displays and so on. A full listing of the switches will be 
given later in tliis chapter after we have talked about the 
coimnands with which many switches are so closely associated. 

In all, twenty-six aspects of system functioning are controlled 
by switches. Of these, approximately twenty aspects can be 
placed under control of the user or of the system itself by 
adjusting the appropriate switches. (Ihe remaining switches are 
concerned with the user's goal, certain d.' splay options and the 
like, and are not relevant to the control issue per se.) 

A switch may have two or more "settings." Complete control of 
his owTi path through the information can be put in the hands 
of the user by setting all switches toward the free end of the 
continuum. On the other hand the system can assume complete 
control of selecting displays if the switches are set toward 
the restricted end of the corttinuim. Numerous patterns of 

shared control can be achieved by the way the switches are 
adjusted. 



continued on next page 



DOMINANCE CONTROLS, continued 



(continued) 

Control 

Switches 



Control 

Levels 



Not all switches are relevant for every mode; for instance, 
control dimensions for initial learning are determined by twenty or 
so switches, but for the review mode only eleven switches are 
involved and for the briefing mode six switches are sufficient 
to define different control aspects. 

The variety of switch settings makes it possible to give enor- 
mous freedom to reference and browsing users and to retain 
various degrees of control over some initial learners. 

The present version of the sequence generator design calls for 
these switches to be set initially by a training supervisor, 
who hopefully will take into account the background and prefer- 
ences of the user when making the settings. 

Later versions of the generator envision the switch-setting 
task performed by a series of internally stored algorithms 
which react to data obtained in a user-con5>uter dialogue at the 

beginning of a session and to stored data from previous sessions 
with the user. 



In this version where switch-setting is carried out by the 
supervisor, the control switches, each with two or more set- 
tings, are too numerous for the supervisor to cope with in a 
reasonable time span. Therefore, as a temporary measure, we 
have made iq) for the supervisor sets of switch-setting patterns 
to serve as guides for placing a client somewhere on the continuum 
ranging from total user control to total system control. From 
these sample patterns, demonstrating how to set up different levels 
of control, the supervisor can select the one best suited for his 
client. Then, working from this basic pattern, he can alter the 
relatively few switch settings that will make it serve the client 
more effectively. 

In addition, some of the switches may be reset within a session 
as a result of the user's behavior (e.g., errorless performance) 
or at the user’s request. 

For initial learners, recommended switch patterns exemplify five 
"control levels," ranging from the level at which all sequencing 
decisions are made by the system to levels with increasing free- 
dom for the learner and on up to the fifth level where he is in 
control of all sequencing decisions. 



continued on next page 
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DOMINANCE CONTROLS, continued 



For reviewers, on the other hand, fewer levels need be illus- 
trated to make the supervisor's task more manageable. 

Throughout this report we v;ill frequently use the term "control 
level" to refer to the relative degree of freedom e^cperienced 
by a user in controlling sequencing. The term is not meant to 
i^ly a uniform pattern of switch settings for all users at a 
given "level" but only a relative position on a continuum. 

While users at a "restricted level" of control will have quite 
similar switch settings and those at the "free level" will have 
most of their switches set alike, the pattern in each case 
represents an individual prescription made by the supervisor 
for the particular user. 

The phrase "balanced level" is used sometimes to indicate switch 
positions that distribute the tasks of sequencing about equally 
to the user and to the system. Obviously many different switch 
patterns could qualify a user for such a designation. 

We shall come back to these relative control levels in the next 
two chapters where we see how . the generator responds to the 
individual user in the different modes . 



Comment The kinds of information that influence the selection of a 

control level for each user and the kinds of data that the 
sequence generator draws on for its work are described in 
the next few maps. 



(continued) 

Control 

Levels 



DATA BASES USED BY THE SEQUENCE GENERATOR 



Introduction In order to accomplish its work the sequence generator must have 

access not only to the information blocks themselves but also to 
masses of information about many aspects of the system — details 
about the courses available, about the current users, about 
scoring each feedback question, about the number of blocks 
available for each map, and so on. 



Table of 
Data Bases 



Such data are stored in a series of data bases. Each base con- 
tains a distinctive kind of information which is described 
briefly in the table below and more elaborately in the following 



NAME 


B^IEF DESCRIPTION OF CONTENTS 

1 


SYSTEM 

DATA 

BASE 


Data pertaining to the total system Ccosmiands , map 
types, switches, etc.), information needed for all 
courses and users, and information about the contents 
of other data bases. 


COURSE 

DATA 

BASE 


Data describing the diaracteristics , content, and 
structure of each course; the information blocks 
’Idiemselves and answer evaluation materials. 


USER 

DATA 1 

BASE 


Permanent and changing data on each user, including 
background data, course performance records, and 
his status within an on-going session (switch set- 
tings and the like) . I 
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THE SYSTEM DATA BASE 



Introduction 



Description 



Exaii5>les 
o£ Uses 



Certain classes o£ in£omiation about the system itsel£, its 
parts, and £unctions are needed £or general use^- Such data are 
stored in the system data base. 



The system data base contains the £ollowing classes c£ infor- 
mation : 

. the name and number o£ each course in the system and the total 
number o£ courses 

. the name and code number o£ each user o£ the system and the 
total number o£ users 

. the names and descriptions o£ all switches, including 
the function of each, the number of values it can have 
and the meaning of each value 

. a store of messages from which the sequence generator can 
select one as a response to an action on the part of a user 

information about the command language and the actions to 
take when each term occurs 

the sequence generator itself (it^ decision tables and pro- 
grams) 

. statistical data across users and courses. 



The sequence generator refers to system data in order to: 

. select the correct course data base for a user 

. determine the next available course number for an author to 
use 



locate the appropriate user data base for a person logging on 

- allow switch settings to be made or changed 

. display a message telling the user he has used an illegal 
command 

. display a message informing the user whether his response to a 
question was correct or not 

provide summaries of system performance for researchers or 
programmers , . 

. provide summaries of user performance for training super- 
visors or researchers. 



THE COURSE DATA BASES 



Introduction 



Des cription 



Hierarchy o£ 
Structures 
Within Each 
Course Data 
Base 



Each course is organized into units. 

Each unit is organized in terms of maps. 

Each jnap is organized in terms of blocks and in terms o£ feed- 
back iquestions as well because these require special treatment 
and evaluation. 

Each of these structures and their contents must be described 
to the sequence generator so that it can select appropriate 
materials to display. 



To organize all this information systematically, a separate 
data base is established for every course in the system. The 
names and locations of these course data bases are specified 
in the system data base. 

Each separate course data base consists of a hierarchy of smaller 
data bases for each structural part of the course. 



Each course data base i s composed of: 

. data pertaining to the whole course , such as: 

. table of contents 
. course index 
. prerequisite chart 
. objectives 
. main points 
. applications 
. overview and summaries 
. san^le blocks and questions 
. difficulty level 
. time estimate for learning 
. final examination. 

. the name, code number and location of every unit data base. 

Each unit data base is compos ed of : 

. data describing the unit to the sequence generator, including 

. major concepts in the unit 
. unit pretest and review questions 
. prerequisite units 



. the names, code number, and location of a data base for 
every map in the unit. 



continued on next page 
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THE COURSE DATA BASES, continued 



Each map data base contains: 

data describing the map, including: 

. map number 

. map type. (concept, procedure, etc.) 
blocks available 
number of exan^les 
possible following maps 

the information blocks themselves in display format 

the number of each feedback question and the location of 
the data describing it. 

Each feedback question data base contains : 

. data about the question, including: 

relative difficulty on a scale from 1 to 5 
. possible answers that can be expected 

the correct answer in the form preferred by the author 
(may be displayed as part of feedback to user) 

. the identity of the information block that is the 

basis for the question (block can be displayed again if 
question missed) 

the number of times the user is allowed to try to 
answer the question. 

. the question itself in display format. 



(continued) 
Hierarchy of 
Structures 
Within Each 
Course Data 
Base > 
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US^R DATA BASES 



Introduction 



Description 



Just as each course in the system has its ovm data base, so does 
each user of the system. It is here that all information per- 
taining to a specific user is stored for use by the sequence 
generator. 



Th® user data base describes the user to the sequence generator. 
It not only holds data about the person himself but it stores 
data about how the system treats him and about his current status 
in interacting with the information bank. Among the items con- 
tained in a user's data base are: 

. background information such as: 

. name 
. age 
. sex 

. current job 
. reading level 
. mathematical level 
. areas of interest 

. achievement scores, attitude data, or other information 
a course designer believes related to infoimation 
sequencing strategies 

. data concerning user's experience with the system, including: 

* what course materials he has used and for what purposes 
Clearning, reference, etc.) 

. perfo-ptiance records in Initial Learning or Reid.ew Modes 
. experience with the command language 
. the control levels at which he has been operating 

. data concerning jhis current interaction with system 
materials, including: 

. the mode in which he is operating 
. name of the course 

. unit pretest scores if his purpose is learning or review 

. goal in terms of breadth of coverage (if relevant to the 
current mode) 

. criterion in terms of level of mastery sought (if relevant 
to the current mode) 

. switch settings (while the system data base contains 
general information about each switch and its possible 
values, the specific value each actually has for this 
individual on this occasion is stored in the xiser's own 
data base). 



continued on next page 
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USER DATA BASES, continued 



(continued) 

Description 



. data recording his path through the materials such as: 

. maps seen and passed 
. units passed 
. units failed 

. temporary data needed by the sequence generator in order to 
keep track of the user’s current status and to conduct the 
session in ^ orderly, meaningful succession of displays, 
commands and responses. They include: 

. the number of blocks he has seen 
. list of maps whose prerequisites he has met 
. the name of the current map' 

. the mmiber of feedback questions failed 
. and many others. 



The devices for managing some of these data classes and for 
keeping them current are in5>ortant aspects of the Learning- 
Reference System. The next map describes more about how they 
function . 



Comment 






( 



VARIABLE DATA SOURCES FOR THE SEQUENCE GENERATOR 



Introduction In the various data bases, we have said, the sequence generator 

finds not only the displayable information blocks themselves 
but also the detailed description of the information to be found 
at different levels of organization — courses, units, maps, 
blocks, feedback questions, and so on. 

And we noted that an important data base is that reserved for 
det^ls about each user — his capabilities, academic history, 
attitudes, experience with the system and so on. These are 
the details that influence the switch settings that determine 
how much control the user exerts over the sequencing. 

But to assemble meaningful sequences of information, the gener- 
ator must keep track of many things during a session. It needs 
other data sources in order to do such tasks as these: 

. determine whether prerequisites have been met 
. assemble blocks for the next map 
. determine how many blocks to show at a time 
check on the legality of a command from a user 
evaluate the response to a feedback question 
. determine whether the current map has been passed 
. re-display a previous map that the user asks for again 

. display a list of maps from which the user may select 
his next topic. 

Several classes of variables have been defined to maintain 
the data needed. 

Description The term variable is used in computer programming to refer to 

several classes of memory locations whose contents are subject 
to change. In general the different kinds of variables (listed 
below) are used to keep track of changing conditions, the 
number of times events have occurred, lists of things that 
have been done or are yet to be accon5>lished and so on. 

pie data held in these memory registers are changed during an 
interaction with a user through the decision tables. The de- 
cision table example at the beginning of this chapter showed 
how flags and strings get changed. The switches, which we 
discussed earlier in reference to the control of information 
sequencing, belong to this grou^) of variables also, but we 
shall save further description of them for a separate chart 
following the command language maps* 

We do not give these tables with the e^ectation that readers 
will need this information, but rather with the idea that they 
might care to browse through to get a feeling for the kinds of 
tasks these variables help with. 
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VARIABLE DATA SOURCES FOR THE SEQUENCE GENERATOR, continued 



VARIABLES OF THE LEARNING-REFERENCE SYSTEM 


NAME 


DEFINITION AND USE 


EXAMPLES 


string 


any combination of 
letters or numbers; 

mostly used to indi- 
cate names of courses 
or maps or blocks. 


CURRENT MAP STRING holds name of map the user 
is working on. 


LAST STRING contains name of the last informa- 
tion block displayed. 


list 


an ordered set of 
strings or of other 
lists; 

mostly used to keep 
track or data about 
displays from maps — 
blocks seen, blocks 
available, blocks 
requested; feedback 
questions seen, 
passed, or failed; 
examples seen; or- 
dering of blocks 
for display, and so 
on. 


EXAMPLES SEEN LIST shows which examples from 
each map were actually displayed. 


USER DISPLAY LIST gives in order the names of 
blocks routinely shoiim to a particular user. 


SUPPLEMENTARY BLOCKS REQUESTED LIST keeps 
track of the kinds of blocks the user asks for 
beyond those on his regular display list. 


[stack 


a list that can be 
read at one end only; 
items are taken on or 
off from same end; 

used for remembering 


LAST MAP STACK saves map names for possible re- 
visiting later. 


LINK STACK saves map names and display condi- 
tions so that a user can return to his place 
after a brief excursion elsewhere. 


iicuuwo <uici uXo 

conditions so that a 
user can return la- 
ter; they also remem- 
ber which units were 
passed or failed 
during pretesting. 


UNITS FAILED STACK keeps a list during unit 
pretesting . 



continued on next page 
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VARIABLE DATA SOURCES FOR THE SEQUENCE GENERATOR, continued 



VARIABLE 


S OF THE LEARNING-REFERENCE SYSTEM, continued 


NAME 


DEFINITION AND USE 


EXAMPLES 


vector 


a sequence of bits 
whose length equals 
the number of maps 
whose status it is 
remembering ; 

mostly used to keep 
track of which maps 
in a course are to 
be shown to a user, 
whic^ ones he has 
failed, passed, and 
so on. 


GOAL VECTOR is as long as the number of maps 
in the course and the only bits that are 
turned on show which maps are recommended 
for the current user. 




MAPS SEEN VECTOR has as many bits as the 
number of maps in the user^s course; all 
bits are off at first but as each map is 
seen, its bit is turned on. 




REVIEW VECTOR operates in the reviewer mode 
to keep track of units t-hose review questions 
were not passed at an acceptable level. 


flag 


flags are usually set 
to Y Cyes) or N (no) 
to show whether or not 


SIL TIME FLAG shows whether this is the first 
pass through the Start Initial Learner Table. 




some conditicn has 
been met; 

they remember what the 
user has done or whe- 
ther the sequence gen- 
erator has gone through 
a decision table be- 
fore; they are also 
important in judging 
feedback questions 
and in showing whether 
commands are legal . j 


GOAL MET FLAG is set to Y when the user has 
seen/passed all the maps indicated on his 
GOAL VECTOR. 




NEXT ONLY FLAG, when set to Y, indicates that 
NEXT is the only command legal for the user 
at that moment. 




GIVE RIGHT ANSWER FLAG, when set to Y, directs 
display of the correct answer to the current 
feedback question. 

\ , 

1 



^ J. W CUJKJKJL^ UiiW XUild 

of these variables will be illustrated as we see how the- 
sequence generator interacts with several types of user. 



; 
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THE COMMAND LANGUAGE 



Introduction 



Definition 



Using the 
Language 



In every machine-based instruction system, the user must have 
some means of communicating his instructions , questions or 
responses to the system. 

The interactive Information-Mapped Learning-Reference System 
has two main ways in which a user can communicate: 

. he can type 

. he can push a button . 

These response modes are used either singly or in combination 
to convey instructions to the sequence generator, such as: 

. show me the next display 
. go back to the last display 
. give me another exan 5 )le 
. let me see the index. 

In actual practice such instructions are given not by such 
time-consuming phrases but by abbreviated commands: next, back, 
exan 5 )le, index. 



The command language consists of a restricted- set of terms 
defining specific instructions that the user can give to the 
computer. There are approximately two dozen commands in this 
set. at the present time plus the names of information blocks 
which function as request commands. 



Some commands are acconqjlished by pushing a conuT!?*T»d button 
(e.g. NEXT, STOP) or by using a button plus a typed phrase 
(REVIEW 6, GOTO 17) in which the number typed refers to a spe- 
cific map. 

The terms ’’command button” or ”pushbutton” are used throughout 
this report but how these specific functions will actually be 
implemented will depend upon the nature of the particular com- 
puter facility. It is quite possible that the command terms 
might be selected not from a pushbutton bank but rather from 
a scope; display of the appropriate terms; in this case a choice 
could be indicated with either a light pen or a teletype key. 
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THE COMMAND LANGUAGE, continued 



(continued) 
Using the 
Language 



Restriction 



Conment 



Typing is mainly limited to giving map numbers, block names, and 
certain letters or nun±>ers in response to multiple- choice questions. 

English phrases or sentences would be typed only: 

. in answer to a test question that requires a constructed 
response 

. in a comment which can be typed in after using the REMARK 
command and which allows the user to record comments and 
suggestions about the materials or the system. 

The entire set of commands is not available to all users. To 
some extent the number that are permitted is a function of the 
degree of control the user is able to exercise over the system. 

In other words, the switch settings and certain flags can make 
some commands inoperable. For exaii5)le, a certain class of 
user might not be allowed to use the command to change courses. 

Restrictions on command use vary also with the mode in which 
the system is operating. For instance, the reviewer is allowed 
to use those commands that enable him to list the kinds of in- 
formation blocks he wants to see regularly; but a person oper- 
ating in the briefing mode would not be permitted that option. 

The briefing mode user can however change courses at any time 
while other classes of clients are sometimes prevented from 
doing this. 



Because commands and switches are very closely associated and 
because they both relate to the important issue of how display 
sequencing is controlled, we devote the next few pages to: 

. a chart- of the commands and their meaning 

. a table of switches and what they accomplish. 
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THE KINDS OF COMMANDS OF THE COMMAND LANGUAGE 



Introduction 



Classifica- 
tion Table 
of the 
Commands 



Some commands are: 

. executed by sinq)ly pushing the appropriate command 
button (e.g., NEXT, BACK) 

put into action by pushing the coonnand button and typing 
some map number (e.g., GOTO 1, REVIEW 6) or block names 

. interactive in that they enable the user to engage in a 
specific dialogue with the system (e.g., REMARK, SWITCH). 

The tables below give the name and a brief meaning of the 
instruction of each of the available commands. 



SIMPLE COMMANDS 


NAME 


MEANING 


NEXT 


Show the next display. 


BLOCKNAME 


Display a particular information block. 


FEEDBACK 


Display a feedback question from the 
current map. 


TABLE OF CONTENTS 


Display the Table of Contents for the 
current course. 


INDEX 


Display the Index for the current course. 


ALL BLOCKS 


Display all of the displayable informa- 
tion blocks from the current map. (Feed- 
back questions are not shown.) 


LAST MAP 


Show all of the displays except feedback 
questions from the map that the user 
last saw before the current one. 



continued on next page 
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THE KINDS OF COMMANDS OF THE COMMAND LANGUAGE, continusd 



Ccontinued) 
Classifica- 
tion Table 
of the 
Commands 


SIMPLE COMMANDS (continued) 


NAME 


MEANING 


BACK 


Show the previous display from the 
current map. 


JUDGE 


Evaluate the user’s answer to a 
feedback question. 


STOP 


Stop the interaction for this session. 


RETURN 


Go back to where the user digressed. 


POSSIBLE MAPS 


Display a list of the maps that the 
user can see. 




APPROPRIATE COMMANDS 


Display a list of the commands that 
would be appropriate for the user to 
eii5)loy at this point. 




BLOCKS AVAILABLE 


Display a list of the information 
blocks available in the current map. 




CURRENT DISPLAY 


Display the names of the blocks on the 
USER DISPLAY LIST. 




SUMMARY 


Show a summary of the user’s work. 



BUTTON-PLUS-TYPE COMMANDS 


NAME 


MEANING 


GOTO”MAP” 


Go to the first display of the map 
whose number is indicated. 


REVIEW ’’MAP” 


Display all of the blocks on the 
MINIMUM DISPLAY LIST of the map whose 
number is indicated. 


DESCRIBE ’’COMMAND” 


Display the map that describes the 
command indicated. 


NORMAL DISPLAY 


Cliange the USER DISPLAY LIST as 
indicated. I 



’y^tinued on next page 







THE KINDS OF COMMANDS OF THE COMMAND LANGUAGE, continued 



(continued) 
Classifica- 
tion Table 
of the 
Commands 



INTERACTIVE COMMANDS 


NAME 


MEANING 


HELP 


Provide suggestions as to how the user 
can better understand the material he 
is working on. 


REMARK 


Allow the user to make a comment. 


SWITCH 


Allow the user to change the value of 
one of his switch settings. 


CHANGE COURSE 


Allow the user to choose another course 
from the list of available courses. 
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SWITCHES USED BY THE SEQUENCE GENERATOR 



Introduction Switches tell the sequence generator, first of all, why the user 

came to the system, what his goal and criterion are Cif applicable 
to his mode) . After that , they mainly specify the degree of 
freedom appropriate for the individual user. Most restrictions 
are defined in terms of limits on the use of the language. 

Just as some commands were not. relevant for some modes, so also 
are some switches inapplicable in some modes. 



The following table will give the reader an idea of the functions 
the switches serve. 



Table of 
Switches 




SWITCH 

NAME 


MEANING 


NUMBER OF SETTINGS 
AND EXAMPLES 


CRITERION 


level of mastery of 
course materials is set 
in terms of feedback 
questions that must be 
answered correctly in 
order to pass each map 


6 se;ttings , including 
. no feedback questions 
shown unless requested 
. must answer two successive 
questions correctly 


GOAL 


the breadth of course 
coverage is set in terms 
of what maps must be 
seen/passed to con 5 )lete 
course 


7 settings, including 
. no goal is set 
. pass major maps 
. see all maps 
. user selects those he 
wants to see/pass 


MODE 


identifies the user’s 
purpose in coming to 
the system 


13 settings, one for each 
of the modes defined earlier 
in this chapter 


GOTO 

LIMIT 


use of GOTO COMMAND can 
be limited to certain 
kinds of maps 


13 settings, including 
. unlimited use 
. use only for maps passed 
. never use (system 
controls sequencing) 


WHEN 

GOTO 


specifies the times 
when use of GOTO is 
legal for the given 
individual 


7 settings, including 
. any-time 

. not during a feedback 
question 
. never use 
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SWITCHES USED BY THE SEQUENCE GENERATOR, continued 



(continued) 
Table of 
Switches 



SWITCH 

NAME 


MEANING 


NUMBER OF SETTINGS 
AND EXAMPLES 




BACK 

LIMIT 


specifies when the 
BACK COMMAND can be 
used 


3 settings, including 
. never 

. only for displays within 
current map 




BLOCK 

LIMIT 


defines the times when 
user can ask for addi- 
tional blocks (except 
feedback questions and 
exanqjles) 


6 settings , including 
. never 

. after required blocks on 
map have been seen 
. use anytime 




CURRENT 

DISPLAY 

LIMIT 


shows when user can ask 
to see which blocks are 
on his current map and 
in what order they will 
be shown 


4 settings, including 
. never 

. not during a feedback 
question 
. anytime 




POSSIBLE 

MAPS 

LIMIT 


specifies when user can 
ask to see the list of 
the maps he has already 
seen 


4 settings, including 
. never 

. net during a feedback 
question 
. anytime 


NORMAL 
1 DISPLAY 
j LIMIT 


shows to what extent 
the user can use the 
NORMAL DISPLAY COMMAND 
to change the kinds of 
blocks and their order 
on his routine display 
list 


7 settings, including 
. never 

. can only re-order blocks 
; can only add blocks 
. can change at will 


EXAMPLE 

LIMIT 


shows when additional 
exanqjles can be re- 
quested 


6 settings, including 
. never 

. only after all required 
blocks have been seen 


EXAMPLE 

ORDER 


shows whether to dis- 
play easier or harder 
exan 5 >les first 


3 settings, including 
. easier first 
. harder first 


FEEDBACK 

ORDER 

L 


shows whether to dis- 
play easier or harder 
feedback questions 
first 


3 settings, including 
. easier first 
• harder first 
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SWITCHES USED BY THE SEQUENCE GENERATOR, continued 



(continued) 

Table of 
Switches • 



1 SWITCH 
NAME 


MEANING 


NUMBER OF SETTINGS 
AND EXAMPLES 


FEEDBACK 

LIMIT 


specifies conditions 
under which user can 
ask for more feedback 
questions 


6 settings, including 
. only after required blocks 
on given map have been 
seen 

. anytime 


MAPS 

PASSED 

LIMIT 


shows whether user must 
do more than meet his 
criterion in order to 
pass a map 


4 settings, including 
. meet criterion only 
. must see all blocks from 
display list 


DESCRIBE 

LIMIT 


shows whether user may 
ask for the description 
of a command 


4 settings, including 
. only when system suggests 
. anytime 


REMARK 

LIMIT 


shows whether user can 
record some comment 
about the system or 
materials 


3 settings, including 
. never 
. anytime 


TEACH 

LIMIT 


defines how much on- 
line teaching of com- 
mands the system should 
provide 


5 settings, including 
. only minimal prompting 
. teach only when user asks 


BLOCKS 

AVAILABLE 

LIMIT 


shows under what con- 
ditions the user may 
ask to see what other 
blocks are stored for 
the current map 


6 settings, including 
. anytime except during a 
feedback question 
. never 


REVIEW 
LIMIT I 


specifies which maps 
the user can see via 
the REVIEW COMMAND 


6 settings, including 
. none 

. any map in current coiirse 
or prerequisite course 


SEE 


records which of his 
switch settings the 
user is allowed to see 


a vector with bits turned on 
for each switch setting user 
can see 


MODIFY 


specifies which switch 
settings the user can 
change 


a vector with bits turned on 
for switches that may be 
modified 



continued on next page 
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SWITCHES USED BY THE SEQUENCE GENERATOR, continuad 



(continued) 
Table o£ 
Switches 



SWITCH 

NAME 


MEANING 


NUMBER OF SETTINGS 
AND EXAMPLES 


DISPLAY 

BLOCKS 


tells how many blocks 
to display at one 
time 


7 settings, including 
. one 

. all blocks except examples 
and feedback questions 


LABEL 

BLOCKS 


shows whether marginal 
labels of blocks are to 
be displayed 


3 settings, including 
. yes 
. no 


COURSE 

LIMIT 


defines when user is 
allowed to change 
courses 


5 settings, including 
. on system suggestion 
. at end of present course 


SET 


shows which switches 
are not yet set 


a vector with tumed-off 
bits showing which switches 
to set before user can 
begin session 

1 
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KINDS OF DECISION TABLES 



Introduction Now that we are acquainted with the various kinds of data that 

the sequence generator must consult before delivering displays 
to the client, we can return to the topic of decision tables 
and consider some of the major functions that they accomplish. 

Description Three basic classes of decision tables in this system are: 

. those concerned with initialization , getting the system 
set iq> to serve the client 

those concerned with display- response interaction during 
the session 

. those concerned with termination of the session or the 
course . 

Each class will be described briefly. 



^^^is-lization When the user begins his interaction with the system, he is 

recognized and either his old data base is activated or a new 
one is established for him. The first example in the chapter 
showed how decision tables could accomplish that. 

Probably e\^en before the user logged on, the siq)ervisor would 
have considered his background data, his purpose, and so on 
and would have set the various switches to establish his de- 
grees of freedom and, if relevant to the mode, to set a 
USER DISPLAY LIST, indicating the blocks from each map type 
are suited to this individual's purpose and the order of 
displaying them. (The user may be allowed the freedom to 
change these.) 

^out a dozen or so decision tables enter into this phase of 
initialization. They check switches, set or unset flags, get 
the user’s data base ready to receive new data, specidfy the 
minim:^ nimiber of examples to be displayed, and so on. A 

sin5>lified exan 5 >le of such a table is given at the end of this 
chapter. 



The fundamental work for which the system was made takes place 
during the cycle of di^lay followed* by user action. The sequence 
generator presents a display; the user then responds by saying 
NEXT, Or by e:q)ressing a desire for some other display or 
information. 

continued on next page 
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KINDS OF DECISION TABLES, continued 



(continued) Two tables carry the main burden of this interplay: 

Display- ^ 

Response . Table 260 Displays 

Interaction . Table 135 Response. 

The first table relies upon several subtables to decide whether 

the user has passed a map or course, or what should be displayed 
next . ^ 



The second table has various subtables to help it decide how to 
react to user's responses— they must identify the nature of the 
response; if it is a command, they must see if it is legal; if 
it is an answer to a question, they must see whether it is to 
be evaluated. In either situation they must find what action is 
prescribed by stepping throu^ the sets of decision tables 
governing these aspects of the interaction. 



Termination When the user stops at the end of a session or a course, various 

housekeeping routines must be taken care of. Certain data col- 
lected about the user need to be saved for future occasions as, 

for exan5)le, when he returns to continue from the point at which 
he stopped. These various chores are done by the third group 
of decision tables. ^ 



Status 

Comment 



At present approximately one hundred two tables have been 
written for these tasks. 



EXAMPLE OF A DECISION TABLE 



Introduction The entries in most of the decision tables of the sequence 

generator are more con 5 )lex than those illustrated at the 
beginning of this chapter. Some tables have twenty or thirty 
columns and rows. Sometimes the cell entries are more varied 
than the sin 5 >le Y, N and X shown earlier — digits may be used 
to indicate a wider range of values. 

For example, a user’s aptitude might be indicated by H, M, or 
L to refer to high, medium or low. In the actions section of a 
decision table, digits might appear to show, for instance, how 
many exaii 5 )les are to be displayed. 

Most of the actual decision tables are unsuitable for illustra- 
tive purposes because they contain special terms or programming 
conventions and are not understandable without considerable 
study or familiarity with the variables that have been described 
in previous maps. 

The table chosen for illustration here does contain some 
special names for computer memory locations and it uses the 
backward arrow to say ’’put something into such-and-such a 
location-” Nevertheless , it may give a feeling for the way 
the operating decision tables work and keep the interaction 
flowing from one table to another. 



Purpose of Table 200, shown below, prepares the system to respond to the 

the Table initial learner or reviewer. If the person has used the system 

before, this table brings up the relevant data that insure 
continuity in its interaction with him. For instance, under 
the heading of ’’fetch restart data,” the system reinstates the 
conditions that existed at the last session — the display 
conditions, the switch settings, the status of vectors, strings, 
lists and so on. 



Legend 



o 

FRir 



It initializes various memory locations. It calls upon 
other tables to perform special routines sucdi as selecting a 
new course. 

It ends by transferring control to tables that will start up 
the displays for a learner or will interview the reviewer to 
find out his objectives. 



The terms in tpper case in the following table are system- 
specific, referring to memory locations, to variables or to 
other tables, which are distinguished also by the table nunber. 
For example, ”D0 680 COURSE CHOICE” means that the computer 
should carry out the instructions given in Table 680, entitled, 
’’Course Choice.” 



continued on next page 
74 




ai 



EXAMPLE OF A DECISION TABLE, continued 



200 START INITIAL LEARNER OR REVIEWER 1^^ 


3 ms 6 7 t 


? 7 49 // 


/2 


CHas he already used the table today?) |n N 

Value of SIL TIME FLAG? I 


nInIn nIyI 


Y Y ? 


Y 


1 (Is he ready to start a course?) IN N 

1 Value of START NEW COURSE user attribute? I 


N N Y Y N 1 


^ N N Y 


Y 


1 (Is he ready to select a new course?) |n N 

1 Value of CHOOSE NEW COURSE user attribute? I 


yIyI- -InIi 


^ Y Y - 


- 


1 (What type of iiser is he?) I 

1 Value of MODE SWITCH? 1 1 7 

1 1 Initial learner I 

1 7 Reviewer I 


17 1 7 ih 


^17 1 


7 


1 (Show that table has been used today.) I- - 

1 SIL -TIME FLAG Y . I 


X x| X xlxh 


r 


- 1 


1 (Set up the cross course data.) |_ _ ■ 

1 DO 250 DATA THERE I 


kIxIx xlxh 




- 1 


1 Fetch user restart data |x X - 


- - - - XX 




- 1 


1 DO 680 COURSE CHOICE 1 - - : 


dxl- -1-1- 


X X 




1 DO 203 CREATE USER RESTART DATA > 


dxlx x|-|- 


XXX 


x| 


1 START NEW COURSE N I 

1 CHOOSE NEW COURSE — N 1- - > 

1 CURRENT MAP STRING NAME OF FIRST MAP I 

ImSPLAY stack USER DISPLAY LIST (TYPE STRING) I 


[XX X - - 


XXX 


x| 


[ goto 260 DISPLAYS |x - }{ 


:|-|x -Ixl- 


X - X 


- 1 


1 GOTO 551 REVIEWER INTERVIEW I - X - 


Ixl- x|-|x 


- X - 


x| 
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CHAPTER 4 THE SYSTEM'S RESPONSE TO THE INDIVIDUAL LEARNER 
OVERVIEW OF THIS CHAPTER 



Introduction Now that the major elements and functions of the Learning- 

Reference System have been introduced, we shall follow the 
system in its interactions with several representative clients. 
This may convey a clearer picture of the workings of the 
system in concrete situations. 



This The topic of individualized instruction is a convenient one 

Chapter aroimd which to organize a fuller description of the system* s 

capacities because it affords opportunities to demonstrate 
some of the options available to users or to system managers. 

Although all sequences are drawn from the same pool of 
organized information blocks, the sequence generator assembles 
quite different configurations of blocks for each individual. 
The factors and processes responsible for the differences in 
information delivered are pointed up in this chapter. 

The stage is set in the early phases of initialization when the 
supervisor and the generator come to various decisions about 
how the new client is to be treated. The first part of the 
chapter deals with that process. 

To illustrate the differential response of the generator to the 
user, the hypothetical cases of three initial learners are 
traced next. A summary map draws together the various dimen- 
sions along which individualization occurs in this system. 



The Next 
Chapter 



Other classes of clients who come with information needs are 
also treated on an individual basis. This chapter describes 
some typical interactions of the system with these users. 
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GEniNG THE LEARNER STARTED IN THE SYSTEM 



Introduction An important part o£ making the sequence generator responsive 

to the individual consists in telling the generator about the 
person the first time he comes to the system. This is accomp- 
lished by a series of tasks assigned at present to the super- 
visor and slated eventually to be performed by the generator 
itself after on-line interviewing and testing of the client. 

In the present version of the system, the supervisor’s work 
is simplified by a computer-directed interview in v;hich the 
generator elicits from' the supervisor the information it needs 
in the form in which it will be used. 



In the following cluster of maps we shall describe the tasks 
involved in starting a new learner. They include: 

. deciding on how control o£ sequencing is to be allocated 
to the system and to the client 

. establishing the learner’s objectives in terms of "the 
breadth of subject matter he wants to cover and the level 
of mastery he wants to achieve 

. specifying the mode (initial learning) and the first 
course the learner wants to see 

. collecting various background data about the learner in 
order to set up appropriate display recipes 

. pretesting ■the client in order to start him at "the 
proper point in the course. 

'■ 9- , 

How each of "these tasks is done is discussed in the next group 
of maps. Fortunately the actual work can be done more quickly 
than it can be described. 

For clients who have used the system before, the system has 
retained their initial data and has added to it the records of 
•their work in the system. Thus for them the initializing 
chores are quickly dispatched. 



Initial 

Tasks 
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ESTABLISHING 



Introduction 



Some 

Relevant 

Factors 



A CONTROL-ALLOCATING POLICY FOR THE FACILITY 



We have a system that can provide varying levels o£ guidance 
for its clients or can leave them free to use the system as 
they will, 

IVhether a level of control will ever be imposed upon users or 
whether they are to choose the guidance level they wish is a 
natter of policy for facility managers to determine. 



Some of the factors that bear upon establishing a policy on 
this issue are: 

. the purpose of the facility — whether it is concerned 
with educational or training objectives 

. whether any job requirements are related to courses 
passed 

. the attitude of the administrators toward teaching 
strategies 

. cost factors for operating the facility — where training ' 
time per person is iii5)ortant, restrictions may be in5>osed 

. customer load— a facility that has queues may need to 
restrict the time allox^ance for each user. 

The various factors interact in affecting the decision reached 
by the managers. The latter may establish a uniform policy 
for the system or a mixed one with differences defined for 
time periods or user classes. 

/ 

In the examples in this chapter, a supervisor presides over 
setting up control levels for the user and we shall assume 
that his actions reflect the facility policy. 
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SELECTING A 



Introduction 



Factors 

Influencing 

Control 

Selection 



CONTROL LEVEL FOR THE LEARNER 



The level of control determines how the learner will interact 
with the system. In this version of the generator where the 
supervisor sets the control switches initially, he must take 
into accoimt certain information about the learner. He may 
find what he needs in the user’s data base if the learner has 
been there before, or he may have to collect it from the 
learner himself or from other sources. 



Some of the factors that enter into the choice of control or 
guidance level are these: 

. user’s history of independence or e^cperience in managing 
own learning program 

. user’s experience with this system and skill with the 
language 

. user’s own wishes 

facility policy (as noted in the last map). 

The learner’s wishes may include such considerations as his 
own time pressures or a desire to expand his acquaintance 
with the system. In the first case, he may have only a short 
time to learn something and, even though experienced with the 
system, he may prefer to be guided along the shortest path 
commensurate with his goals. In the second instance, he can 
be assigned to a guidance level that gives him a chance to 
expand his understanding of commands gradually while he concen- 
trates on his main learning task. 

The amount of control that the user is allowed to exercise 
over the system should be conpatible with. his ability to 
make efficient use of it. It should not put demands on him 
that distract him from his main objective. 

The level selected can of course be changed if the user finds 
it uncomfortable and in any case, switch settings may change 
through the course of a session as a result of the learner’s 
performance or behavior patterns. 



continued on next page 
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SELECTING A CONTROL LEVEL FOR THE LEARNER, continued 



Future 

Plans 



In planning one's own learning program, the amount of freedom 
or self-direction that is optimal is not now known. It will 
probably be found to vary with characteristics of the learner 
and with his stage of development. 

One attractive feature of the Learning-Reference System of the 
^^ture is the opportunities it offers for a practical research 
attack on this problem. With its facilities for allocating 
degrees of control, for measuring learner's progress, for 
recording interactions, and for printing out data summaries for 
many variables, it is well situated to enlighten our understand- 
ing of the control issue. 



How the quality of the learner's experience differs for certain 
control levels will be illustrated presently in exanroles of some 
representative users. First we finish explaining the process of 
preparing the system initially to respond to the user. 



Comment 



DETERMINING THE LEARNER'S OBJECTIVES 



Introduction 



Setting the 
Learner *s 
Goal 



The next in^ortant information the generator needs to know 
about the user is the nature of his objectives. Unless it 
knows these, it will not knovr how much of a subject matter to 
cover in its presentations. 

The learner may come to the system for a variety of purposes. 
He may seek con 5 )lete mastery of the subject area, or he may 
want only an understanding of the main ideas and a knowledge 
of where to find details when he needs them. He may want an 
even more general bird’s-eye view of the subject. 

The sequence generator must be able to bring sequences that 
are appropriate for the objectives of each individual. 



In order to give the sequence generator the specific datum it 
needs, we can arbitrarily define goals in terms of several 
configurations of maps that would serve different objectives. 
Some plausible combinations of maps are: 

, all maps of the course 

, major maps — those presenting the main concepts 
of the course 

, general maps--the previews, reviews, summaries 
and the like. 

The grouping that seems best suited to serving the learner’s 
goal can be selected and indicated to the sequence generator 
by setting the Goal Switch, one of the twenty-six switches 
mentioned in the last chapter. For instance, the switch could 
be set to 2 to indicate the second combination above. This 
selection would also be automatically reflected in the Goal 
Vector which is set up to show the generator which maps in a 
course can be drawn upon for displays. 

The above groupings of maps are examples of the fact that 
different combinations can be earmarked to serve different 
objectives. It would be the responsibility of the course 
designer to tag maps that are suited for different goal state- 
ments, The sequence generator will be guided by the Goal 
Switch setting and the associated Goal Vector to pick out 
all maps with similar tags. 

The Goal Switch could be set as the result of a dialog between 
the learner and the conputer, but for the present version, the 
supervisor confers with the user about his objectives and then 
sets the switch. 
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DETERMINING THE LEARNER'S OBJECTIVES, continued 



(continued) 
Setting the 
Learner's 
Goal 



In the case of those learners who operate near the free end 
of the control continuum, they can set up their own Goal 
Vectors in a dialog with the coii5)uter. Basically what happens 
is that the learner selects from a table of contents the names 
of idle maps he believes he will want to see. The numbers of 
these maps are tagged in his Goal Vector and the generator 
will derive displays from them. 



Setting the The user's goal determines what information he will be exposed 
Learner's to, but how well he must learn that information is specified 

Criterion by a criterion. 



The criterion in the Learning- Reference System is a statement 
of a level of proficiency needed in order to "pass" each map 
shown. This idea of "passing a map" is stated in terms of the 
number of feedback questions that the user must see and/or 
answer correctly. 



Here again a switch is used to select a level that is desirable 
for the specific individual's situation. The levels range 
from virtually no requirements at all for passing a map other 
than just seeing it, to the stipulation that two correct 
responses in a row are required before the user can go on to 
another map, (The reqxiirement to pass feedback questions 
carries with it the inplication that remedial sequences will 
be siitpplied until criterion is met.) 

As the table of switches in the last chapter shows, the 
criterion switch has six settings at the present time; others 
could be added and the settings could be made to change in the 
course of a session as a result of the learner's performance. 
Research in this area will help in formulating the algorithms 
that can set the criterion automatically in the future. 



Comment Taking the goal and criterion settings together, we have the 

means of making the generator responsive to the individual's 
learning objectives. 

A person, for exanple, who was preparing for an inportant 
closed— book examination might speci:fy that he wanted to cover 
all maps of the course and to pass each by correctly answering 
two successive feedback questions. 

On the other hand, the student who wanted only an understanding 
of the main ideas in order to prepare for an open-book test 
might ask for major maps only, and he mi^t set a criterion of 
one correct answer to a feedback question merely to verify his 
understanding of the related map. 
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DETERMINING THE LEARNER'S OBJECTIVES, continued 



Ccontinued) Another user mi^t want only a general picture of a subject, but 

Comment one that was rather more detailed than that provided by the 

briefing mode. In this case he could ask that the generator 
show him all general maps but no feedback questions unless 
instructed to do so by the user. 



PREPARING THE USER DISPLAY LIST 



Introduction 



User Data 



Defining the 
User Display 
List 



Since each learner comes to the system with his own objectives 
and with his unique personal characteristics, the system 
should be able to .create for him a pattern of information 
blocks that is tailored to his needs. The generator would 
not display for him all the stored blocks belonging to a given 
map but would select only those that serve the user’s purpose. 

To do this the generator sets up for each client a list of the 
blocks that it will routinely show from each type of map. This 
list, the User Display List, is prepared after the generator 
consults background data in the user data base or collects the 
information itself. 



For each course, the course designer may specij^ certain 
personal data that will influence decisions about building 
displays for the individual. These data may be obtained via 
questionnaires, proficiency tests, miniature situation tests, 
3nd attitude tests and so forth, either given on-line 
or off-line. The nature of these tests, the scoring of them 
and the decisions to be taken for given score ranges will be 
specified by the course designer. 

For exaa 5 >le, if the course were one in the math area, the 
designer mi^t supply an attitude— toward-math scale and pre- 
scribe that students below a certain score be given designated 
sets of blocks from each map in order to allay their fears of 
the subject ^ea. But for learners: with extensive training and 
Proficiency in mathematics, he might prescribe a mi -ni Tmnn 
display list consisting, for exan 5 >le, of only definition block, 
diagram and notation blocks, plus one exau^jle with a hi^er 
difficulty rating. 

All such prescriptions are recorded in the course data bases, 
piey are elicited from course designers in coii5)uter-directed 
interviews or via printed questionnaires . 



The User Display List contains the names of specific informa- 
tion blocks in the order in which they will be displayed for 
the particular user. The names always refer to the current map 
type (e.g., concept map, procedure map). 

As the starting point for preparing a User Display List, the 
generator builds upon the Minimum Display List, a list that is 
defined in the system data base as constituting the mi -ni mnm 
display allowed from the given map t)pe. For example, let us 
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PREPARING THE USER’ DISPLAY LIST, continued 



(continued) 
Defining the 
User Display- 
List 



say that for a concept map the Minimum Display List consists of 
. name 

. definition/description 
. formula 
. not at' 

. example. 



(If a block named does not exist for a given map, it is 
simply ignored when the display list is used.) 

After considering the current client’s backgroimd data, the 
generator adds blocks to establish the User Display List: 



. name 

introduction 

. definition/ description 
. example 
. formula 
. notation 
. exan5>le. 

The user’s data base may show 
has had a tendency to request 
example; in such a case, that 
display list. 



that in past sessions this user 
display of ’’Use" blocks, for 
block may be made part of his 



Use of The generator consults the User Display List to determine: 

the Display 

List , which blocks to display and in what order they should 

be shown 

, whether all blocks have been displayed when the user 
.presses certain command buttons (under some control 
settings, commands will not be accepted until the 
recommended blocks have been shown). 



Changing The User Display List is a list of recommended blocks but, 

the User excep-t for the client from the most restricted control level. 

Display List there are many possibilities for modifying it. We are con- 
cerned now only with establishing it in the first place. As 
we will see presently, the user’s e^erience with it and his 
personal preferences may mold it much more precisely to satis- 
fying his individual information requirements. 




SETTING UP DISPLAY CONDITIONS 



Introduction Before the learner finally begins his interaction with the 

information base, there remain a few additional chores. For 
one thing, several more aspects of the displays have to be men- 
tioned to the generator. And it has to be told where to begin 
building displays. 



Display 

Details 



Starting 

Place 



Comment 




For both exan5)le blocks and f'^edback questions, the supervisor 
can recommend the difficulty level from which the items are se- 
lected for display. As the table of switches in the last chapter 
noted, he chooses whether easier or harder items are shown first. 
Under the freer control conditions, the learner himself makes 
these choices. 

In an Information-Mapped book, the marginal labels for the 
information blocks are convenient for reference purposes. There 
is also some research that suggests that to be alerted to the 
nature of upcoming information facilitates learning. Neverthe- 
less individual preferences or research purposes make it advan- 
tageous to be able to suppress or display the label. The Label 
Blocks Switch permits the choice and its setting is left up to 
the user (except in controlled research) . 

One more ii^ortant switch remains to be set — that telling the 
generator how many blocks to display at a time. The present 
switch settings can specijfy from one at a time to all blocks 
at once (from the User Display List) , and may be selected either 
by the supervisor or by the learner, depending iq)on how the 
control level is set. 

In actual inq)lementation in a con5)uter facility, the display 
options will become much more varied and inQ)ortant. Since these 
are so closely tied to specific hardware, this version of the 
generator is concerned with nothing more than the number of 
blocks to display at a time. 



Each initial learner beginning a new course takes a pretest 
(unless he is operating under the freest control conditions) . 

This test allows the system to determine the current state of his 
knowledge about a subject. 

The sequence generator uses the results of the pretest (which are 
stored in terms of unit scores) to determine on which unit the 
user should begin. Those units that have been passed in the 
pretest are bypassed by the learner but maps from them may be 
called if he wishes to check on his grasp of some idea in them. 



To illustrate how the interaction with the system differs for 
individuals whose initial settings are quite different, we will 
follow the, sessions of three imaginary learners in the next few 
maps . 




A LEARNER OPERATING AT THE RESTRICTED CONTROL LEVEL 



Introduction 



Initial 

Setx:^ 



Suppose our first fictional learner, Ted, has been assigned to 
the most restricted level of control in which the generator 
makes the sequencing decisions. We shall see how the system 
was initialized for him and then describe the kinds of things 
he can and cannot do in a learning session. 



Ted came to the system for a course in probability. Through 
on-line testing and interviewing, the following facts about 
him emerged that led the supervisor to set up the system in 
this way: 



These Characteristics Led. . . 


► 


...To These Settings 


. dependent, docile learner 
. has never used this systen 
. is in a hurry 


0 


Control Switches to Restricted 
Position except Feedback 
Limit Switch is set to allow 
any feedback question 
requests 








. wants only main ideas 
. high level of mastery 


o 


Goal Switch: See Major Maps 
Only 

Criterion Switch: Pass 2 con- 
secutive feedback questions 
per map 








. math aptitude score=85% 

. lacks confidence in math 
ability 

. little interest in math 
. asks one example to be 
shown before definition 


0 


User Display List: Introduc- 
tion, Exan 5 >le, Definition, 
Notation, Diagram, Comment, 
Exan5)le 

Exan 5 )les and Feedback Ques- 
tions: easier ones first 








. prefers display of two 
blocks at a time 


0 


Block Display Number: Set at 2 
(exclusive of Map Name block 
and Related ^fa.ps block) 


. prerequisite test passed 
. all unit pretests failed 


0 


Starting Map: first major map 
of Unit 1. 



c^i^nued on next page 



A LEARNER OPERATING AT THE RESTRICTED CONTROL LEVEL, continued 



Explanation Ted is assigned to the restricted control level because he is 

of Settings unfamiliar with the system and in a hurry. If he had more time 

and expected to use the system often, some switches could have 
been set to allow him more leeway in using commands and to enable 
the system to proB5)t him on their proper use. 

Or the supervisor might have urged him to take first the short 
course on operating the system. However, since he has little 
time, the system is now set up to guide him through the materials 
as swiftly as possible to achieve the goal he has indicated. 

Because he lacks confidence in his math ability in spite of 
creditable test scores, the generator follows a prescription 
stored by the course designer and sets up a User Display List 
containing both Introduction and Comment blocks. These often 
supply interesting context and extra learning aids that may 
increase his confidence. Two exan5)les were prescribed for the 
same reason. 



The Learning The session starts with a very brief introduction to the system. 
Session explaining the block structure of the displays and the use of 

the few basic commands he will need. 

The course begins with the first major map of the first unit. 

The first display shows the Name and' Introduction blocks and 
an easy exan5)le. 

After reading these blocks, Ted has only a few options -- he can 
push: 

NEXT button to replace the present display with the next two 
blocks 

APPROPRIATE COMMANDS button to see what commands he rar> use 
now 

. STOP button to bail out of the session. 

If he makes the usual response of pushing NEXT, the displays will 
continue to appear two at a time until he has seen the blocks 
from that map that were prescribed on his User Display List. 

Then the feedback questions come up one at a time until he answers 
two in a row correctly (according to his Criterion). When he 
answers incorrectly, the generator follows the directions stored 
by the author in the feedback-question data base — it may show 
again the block from which the missed question was derived or it 
may take some other recommended action. 



continued on next page 
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A LEARNER OPERATING AT THE RESTRICTED CONTROL LEVEL, continued 



(continued) 
The Learning 
Session 



Individual 

Treatment 



Now a new option becomes available to the learner — he may ask 
for more feedback questions if he wishes, because the Feedback 
Limit switch setting permits this. 

From the second map on, he can use the LAST MAP command to bring 
up again the last map he saw. This command and NEXT are the 
two commands that enable him to move the displays — but the 
distance he can move them is very limited. He can back up only 
to the just previous map and he can move ahead through the 
blocks of his current map. Until the required feedback questions 
are correctly answered, he cannot call up the next map. 

These restrictions keep him on the path prescribed for him inir 
tially and prevent him from side excursions. For some educational 
purposes this may not be wise but for certain training objec- 
tives it is advantageous. 

as the session proceeds, Ted consistently answers feedback 
questions correctly, the pnerator may change to the more diffi- 
cult questions and may omit the second exanq)le from the User 
Display List. 

Ted continues through a session with few variations except that 
he can ask for help with the HELP button and. he can use the 
command to get a summary of his session showing how many maps or 
units he has passed and how much remains to be done. 

When he wants to stop, he pushes the STOP button j the generator 
stores away all data concerning his display conditions, his 
place in the course and relevant lists, stacks, strings, flags. 
These are kept in Ted’s assigned data base so that on the next 
occasion he can resume where he left off. 



Ihider these restricted conditions, the learning experience feels 
to Ted very much like a session of coii5)uter-based programmed 
instruction. He was allowed a handful of extra choices perhaps, 
but otherwise he was locked into sequences assembled by the 
generator. 

But as a matter of actual fact, the sequences Ted saw were pre- 
scribed for him as an individual — he was not pushed through, 
in programmed- instruction manner, a standard series shown to 
everybody except for occasional remedial side trips. Itts sequen- 
ces were tailored with an eye to his objectives, his attitudes, 
his prerequisite skills, his personal preferences, and even to 
his present situation — a specific information need and a time 
limit. 



continued on next page 



A LEARNER OPERATING AT THE RESTRICTED CONTROL LEVEL, continued 



(continued) 

Individual 

Treatment 



In addition, before Ted returns for another session, the si:5)er- 
visor or curriculum specialist may review the records of his 
interaction with the course materials and may insert into his 
data base changes in switch settings or display conditions that 
will make the pattern of the next session even more suitable 
for his current status. In a later version of the system, such 
adjustments will be made automatically by the generator. 

Another student assigned to this same restricted level m-i ght see 
a markedly different set of blocks from the same course. A 
person who was proficient and confident in mathematics, for 
instance, might have a short display list consisting of the mini- 
mum allowable blocks and he might have no requirement to answer 
feedback questions, depending i:q)on his objectives in taking the 
course. 
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A LEARNER OPERATING AT A BALANCED CONTROL LEVEL 



Introduction 



Initial 

Setup 



Our second imaginary learner, Dick, has been assigned to the 
control level at whidi some sequencing tasks are performed by 
the system while others are assigned to Dick. It must be remem- 
bered that the setup for Dick is only one of the many quite dif- 
ferent switch-setting patterns that are possible at this level. 

Dick also comes to learn probability but he seeks just a general 
understanding of the major ideas for his own reasons. He has 
taken three previous courses on this system. His stored data 
show that he has a good background in mathematics and is not 
intimidated by it. He is a moderately dependent learner but 
aspires to be more self-directed. In the previous courses with 
the system, he worked at a fairly restricted control* level because 
he did not know the command language well. He has completed the 
on-line pretest for probability and has passed unit 1 at the 80% 
correct level, but for other units he passed less than 50% of 
the questions. 

In conversation with the supervisor, Dick ejqilains his objectives 
with regard to probability materials and his wish to assume 
more control over his learning experiences. 

In view of these various data about Dick, the supervisor adjusts 
the system initially to enable him to: 

. make his own Goal and Criterion setting 

. specify whether easier or harder examples and feedback ques- 
tions should be displayed first 

. indicate how many blocks he wants displayed at a time 

. set his own User Display List naming the blocks he wants 
to see from each map. 

Dick decided to set his Goal to "see all maps" and his Criterion 
to p^s one of the more difficult feedback questions. Because 
of his switch settings, he is free to change these at any time 
if he is not satisfied with them. • 

The supervisor, in view of Dick*s learning history, sets certain 
command switches to limit his ability to junqj about at will in 
the course materials — he can move forward only to naps for 
which he has passed the prerequisite maps. Whenever several 
maps are eligible, he may select the one he wants to take up next. 

continued on next page 



A LEARNER OPERATING AT A BALANCED CONTROL LEVEL, continued 



Ccontinued) He may use all commands but a few cannot be used when a response 
Initial to a feedback question is expected. 

Setup 

The system is also set t:^) to tutor him on commands and strategies 
in order to increase his skill and confidence in handling the 
system. 



Actions He During the learning session, a few of the more in^ortant actions 
Can Take that Dick can take are these: 



. use the GOTO command to see any map for which he has met 
prerequisites 

. use the RETURN command to go back to the place where his 
digression began 

. back up to a previous display from the current map 

. select a topic from the Possible Maps List except when asked 
to answer a question 

. ask for additional blocks to be shown from current map 

. add, delete or reorder the blocks on his User Display List 

. ask for e3q)lanation of commands 

. record remarks about the material, the system, his progress 
and the like 

. consult the table of contents and the index. 



Actions He 
Cannot Take 



Few things are forbidden to a user at Dick*s level except that in 
his case switch settings specify that: 



. he cannot jun^) ahead to maps he is not prepared for 
. he cannot change a few switches 

. he cannot avoid displays of feedback after answering feedback 
questions 

. he cannot change courses (a decision of the si:q)ervisor*s). 

An exaD 5 )le of Dick*s interaction with the system during a learning 
session is charted on the next map. 
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SAMPLE SESSION OF A LEARNER AT A BALANCED CONTROL LEVEL 



Introduction 



Initial 

Setup 



Format 



To give an inpression o£ how a session might proceed, we 
present a san^le o£ the interaction between Dick and the 
coa^uter. The exan 5 )le is constructed to show some o£ the 
variety o£ actions Dick can take; in actual practice one 
would hope that he would pursue a less erratic course. 



To recap the display situation set up in the preceding map, 
mainly by Dick's own choices: 



Goa.1: 
Criterion : 
User Display List; 

Examples and 
Feedback Questi ons ; 
Blocks at a Time; 

Course ; 



See all maps 

Pass one Feedback question per m^p 
Definition, Notation, Diagram, 

One Exanple 

Display harder ones first 
Display two 

Introduction to Probability. 



To save space, the information blocks themselves are indicated 
only by their labels in parentheses; the user's answers to 
feedback questions are similarly indicated. 

Actual map names and displays of conputer remarks to Dick are 
regular text format. The commands Dick uses are 

given in upper case. E3q)lanations are added on the right when 
^propriate. 



continued on next page 



SAMPLE SESSION OF A LEARNER AT A BALAf^CED CONTROL LEVEL, continued 





SEQUENCE 


EXPLANATION 


Display: 


Events. Map 25 
(Definition block) 
(Notation block) 


The first map from Unit 2 is 
shown (Unit 1 having been passed 
in the pretest) . 

B lodes indicated on User Display 
List are shown two at a tine,. 


User: 


NEXT 


' 


Display: 


Events. Map 25 
(Example block) 
(Related Maps block) 


Map 25 has no Diagram block, 
generator displays next available 
block. (Feedback questions are 
not shown at same time as blocks.) 
Related M^s block, a local index 
showing location of main concepts 
relevant to the given map, is 
automatically displayed at end of 
eadi map. 


User: 


NEXT 




Display: 


(Feedback Question) 




User: 


(Answer) 

NEXT 




Display: 


Your answer is correct. 
Push NEXT to continue. 


The criterion of one correct 
answer has been met; Map 25 is 
recorded as "passed,” 


User: 


GOTO 17 


Dick wants to review a concept in 
Unit 1 which he had passed. 
Related Maps listing helped him 
locate it quickly. 


Display: 


San^le Space, 

San^jle Points. Map 17 
(Definition block) 
(Notation block) 




User: 


RETURN 


Dick learned what he wanted from 
the first two blocks, stops dis- 
play of remaining blocks by asking 
to return to his previous 
place . 


Display: 


Enpty Event. Map 26 
(Definition block) 
(Notation block) 


Since Dick had passed Map 25, the 
computer continues with the next 
scheduled map. 




continued on next 


page 
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SAMPLE SESSION OF A LEARNER AT A BALANCED CONTROL LEVEL, continued 





SEQUENCE 


EXPLANATION 


User: 


NEXT 




Display: 


Empty Event. Map 26 
(Example blodc) 
(Related Maps block) 




User: 


BLOCKS AVAILABLE 


This command gets the generator 
to list the names of all blocks 
available on the current map. 


Display: 


Enpty Event. Map 26 
Blocks Available 
Introduction 
Definition 
Notation 
Comment 
Example 
Non>Exanple 


• 


User: 


COMMENT 




Display: 


Empty Event. Map 26 
(Comment block) 




User: 


NORMAL DISPLAY DEFINITION 
NOTATION DIAGRAM EXAMPLE 
COMMENT 


User wants to make Comment blocks 
a regular part of his Display 
List in the order shown. 


Display: 


New Display List recorded. 




User: 


NEXT 


■ 


Display: 


(Feedback question. Map 26) 




User: 


(Answer) 

NEXT 




Display: 


No:, that is not correct, 
(Notation block) 
(Feedback question) 


The block on which the question 
was based is displayed again. A 
different feedback question on the 
same point is displayed as well. 


User: 


(Answer) 

NEXT 




Display: 


That is correct now. 
Elementary Events. Map 27 
(Definition block) 
(Exanple) 


Map 26 recorded as passed. 
Notation and Diagram blocks not 
available for this short Map 27, 



JQJ continued on next page 



SAMPLE SESSION OF A LEARNER AT A BALANCED CONTROL LEVEL, continued 



User: 



SEQUENCE EXPLANATION 

NEXT 



Display; 



User; 



Display; 



User; 



Display; 



User; 

Display: 



User; 



Elementary Events. 27 
CComment block) 

(Related Pages block) 

GOTO 34 



Sorry, you have not 
passed prerequisites. 

POSSIBLE MAPS 



Without passing Map 27, Dick 
asks to junq) ahead to another 
topic . 



This command will get Dick the 
list of those maps that he is 
eligible to call up. 



The following maps are those 
for which you have met all 
prerequisites. Asterisks mark 
those you have already seen 
and passed; 

Number Map Name 

25 *Events 

26 *Eii5)ty Events 

27 Elementary Events 

28 The Ihiion of Two or 

More Events 

29 Elementary Events Form 

Other Events 

30 The Intersection of Two 

or More Events 

GOTO 30 

The Intersection of Two or 
More Events. Map 30 
(Definition block) 

(Notation block) 

FEEDBACK QUESTION Since Dick has the concept, he 

forestalls display of the other 
blocks by asking for a question. 



Dick prefers to take up this topic 
next rather than the one the 
generator chose for him originally. 



continued on next page 
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SAMPLE SESSION OF A LEARNER AT A BALANCED CONTROL LEVEL, continued 



. 


SEQUENCE 


EXPLANATION 


Display: 


(Feedback question Map 30) 




User: 


(Answer) 

NEXT 




Display: 


Right. Continue. 


Map 30 joins Maps Passed List. 


User: 


STOP 




Display: 


Your current status is: 
Passed: 25,26,30 
Other 

Possible Maps: 27,28,29,31 
This record will be kept for 
your next session. 


Status and all display conditions, 
lists, flags, etc., stored for 
next occasion. 
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A LEARNER OPERATING AT THE FREE CONTROL LEVEL 



Introduction Our third imaginary learner is Henry, who is an old hand at man- 
ipulating this system as well as being a self-reliant person 
accustomed to setting up his own objectives and a plan for 
achieving them. The supervisor has intelligently adjusted the 
system so as to give Henry carte blanche. Let us see what help 
the generator can offer Henry. 



Initial 

Setiq) 



Learning 

Session 



Any learner at this free level of control can use any command 
and set any switch he wishes at any time. Whether he wants to 
set up a Goal and a Criterion is con 5 >letely up to him. 

Henry’s situation is this: he wants to prepare for the Graduate 

Record Examination. Although he is well prepared in mathematics, 
he knows little about probability. 

His first move is to take on-line the pretest for the Introduc- 
tory Probability course. Units 1 and 2 are passed at a 90% 
level which satisfies him. His concern now is to learn more 
about units 3 to 5 . Rather than set any Goal or Criterion or 
display conditions now, he begins the learning session in his 
own way. 



Henry changes the Mode Switcli to the briefing mode setting. He 
then quickly runs over the main points of the probability course, 
its objectives and the course summary. 

Once he has the big picture of what the course is about, he re- 
turns to the Initial Learner Mode by changing the Mode Switch. 



Now he uses the goal and criterion functions of the system to 
design his own course. He calls for the table of contents, 
checks off the names of maps that he wishes to see. These go 
into his Goal Vector so that each of them comes . up in order 
unless he chooses to deviate from his plan. 

Because he is conpetent in mathematics, he sets no "passing” 
criterion but merely asks to be shown two difficult feedback 
questions for each map. 

Also because of his good background in mathematics, he sets up 
for himself only the Minimum Display List: Definition, Notation, 

Diagram, One Exaa 5 >le. 

He sets the switches to display all blocks at a time, keeping 
the labels attached. 



continued on next page 
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A LEARNER OPERATING AT THE FREE CONTROL LEVEL, continued 



(continued) 

Learning 

Session 



Now he begins study o£ the displays from Unit 3, using the NEXT 
command to move the maps along on the scope. At any time he is 
free to depart from the sequence he has established. He may 
pursue related ideas to previous units or go ahead to unseen and 
unpassed materials . 

He may call up additional blocks any time he wishes or he may 
design a new User Display List. 

If, on reading feedback questions, he feels uncertain about the 
answer, he can give his response and by use of the JUDGE command, 
can get feedback concerning his correctness. (Ordinarily, he 
does not need feedback for he knows when he understands.) 

He may call for the next unit pretest, if he likes, to satisfy 
himself about his progress in that area. His scores have no 
automatic effect on his procedures — he can adjust his strategy 
or not, as he wishds. 

If he forgets the meaning o£ certain terms in the command lan- 
guage, he can ask for help at any time. He may even leave the 
probability course and revisit the course on how to operate the 
system. 

He is free to record remarks about any aspects of the materials, 
of the system, or of his own progress. 

He may discard his systematic approach to the unpassed units of 
the course — he might, for instance, call up the feedback ques- 
tions for maps picked at random from the table of contents. Only 
when he is ignorant of the answer would he call the informa- 
tion blocks from the map in question. 

After a series of experiences of that sort, he may resume an 
orderly inspection of the materials in each unit. He is free to 
follow whatever strategy appeals to him at the moment — even to 
choosing a completely guided tour of the subject area, such as a 
learner at the most restricted level might receive. 



The generator is designed to help the self-reliant learner in 
many ways . The system is also designed to in^rove its services 
in response to user’s remarks and suggestions and in response to 
research summaries of operational data. 

Future plans for the system include the addition of special ma- 
terials and simulation games designed to encourage students to 
become self-reliant managers of their own learning experiences. 
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SUMMARY OF THE SYSTEM'S RESPONSE TO THE INDIVIDUAL LEARNER 



Introduction Hie initial learner in the Learning- Reference System is treated 

as an individual because the sequence generator acts on many 
kinds of data that are specific to the person. 

This table summarizes those factors about the learner that make 
a difference in system functioning. (Control switch settings 
modify the effect of some factors.) 



THESE FACTORS. AFFECT. . . . THESE SYSTEM COMPONENTS 

I. User’s Objectives 



. pui^ose in entering system — to learn: 
. general nature of subject area 
. entire subject matter 
. main ideas of the subject 



» Goal Switch 



. proficiency level 

. thorough masteiy for long-term retention 
. general understanding 
. other 






Criterion Switch 



II. User’s Background Data 

. independence and e^qjerience in 
planning own learning e:qjeriences 

• proficiency in using this system 
and the language 

. user’s wishes 

. experience in subject area in terms of 
courses and their difficulty level 

. proficiency in subject area in terms of 
prerequisite test scores 

. attitude toward subject area in terms 
of questionnaire scores 

. other cognitive and motivational aspects 
judged relevant by course designer, such 
as reading level, intelligence level 



Control Level 



User Display List 
Number of Examples and 
Feedback Questions 
Their Difficulty Level 



stored records of past command usage or 
other behavior tendencies 

user preferences 



pretest scores 



} 

} 



Display of Block Labels 
Number of Blocks Shown 
at a Time 

First Map Display 



er|c 



0 






SUMMARY OF THE SYSTEM'S RESPONSE TO THE INDIVIDUAL LEARNER, continued 



III . Learner *s In-Session Choices 

(Depending on Control Level) ^ 

• request additional blocks for current map 

. request more feedback questions/examples 

. modify User Display List by adding, 
deleting or re-ordering blocks 

. return to previously seen or passed maps 

. call up table of contents and index 

. select next topic from list of maps ^ Decision Table Usage 

whose prerequisites have been met '* Switch Settings 

. e:q)lore course materials at will 
(free control level only) 

. enter comments into the system 

. request session summary 

. ask for help of several kinds 

. request description of command use 

. request suggestions of possible next 
commands 

. modify display conditions (number of 
blocks, label display) 

• suppress display of answer evaluation 
(feedback) 

. modify switch settings 



THESE FACTORS 



AFFECT 



THESE SYSTEM COMPONENTS 



IV. Learner *s Response Patterns 
feedback failure rate 



Number of Worked Examples 



User Display List 



Number of Feedback 



Questions 



. block order preferences 
. additional block requests 




Remedial Sequences 



User Display List 



command usage record 



Instruction by System 
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CHAPTER 5 THE SYSTEM'S RESPONSE TO INDIVIDUAL USERS 
OF OTHER CLASSES 

OVERVIEW OF THIS CHAPTER 



Introduction The Learning-Reference System is equipped to give custos- 

tailored service to users other than initial learners. In order 
to do this it h^ special features that serve a given user class 
nut it also relies heavily on the mechanisms that generate dis- 
plays for the individual learner. 



This chapter explains how individualized treatment is accomp- 
lished for clients who come for reference, review or briefing 
purposes, A few of the special aids to supervisors, 
writers /editors, and researchers are described also. 



This 

Chapter 
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THE REVIEWER MODE 



Introduction One o£ the primary purposes o£ the Leaming-Re£erence System is 

to help the user review materials that he has previously learned. 
In many respects the operating mode that caters to reviewers 
resembles the Initial Learning Mode and it does use some o£ the 
same mechanisms. 



Description The Reviewer Mode is designed to give the user a guided review o£ 

course materials or to help him review in whatever way he wishes. 
Whether or not he is aliov;ed a choice depends upon the SL^ervisor. 



The 

Guided 

Review 



The guided review, whether chosen by the user or assigned to him, 
takes into account various kinds o£ in£ormation about the 
reviewer and his objectives in much the same way as was done £or 
the initial learner. 

The reviewer may come £or a number o£ reasons. He may want to go 
over some in£ormation he eiq)ects to need in his occupation . Or 
he may have to instruct others and wish to re£resh his memory 
£irst. Or, as a member o£ the academic world, he may £ind review- 
ing useful before taking exams or writing term papers. He may 
even have some humble purpose such as killing time. 

His objectives and certain other personal characteristics enter 
into the supervisor’s deliberations when he sets up the system 
to serve this person. As with initial learners, the supervisor 
considers the user’s experience in operating this system, his 
self-reliance, and his wishes in regard to planning his own review. 

Sets of review questions for each course unit serve as the basis 
for certain sequencing decisions for users in the guided review 
condition. The reviewer can be required to pass a specified per- 
centage of unit review questions, depending upon his objectives. 

If he is reviewing in preparation for an exam, he might want to 
pass 90 per cent of the review questions before leaving the system. 
Units that are not passed at the specified level are shown again. 

A guided review session very much resembles a restricted initial 
learner’s session* in that the reviewer has: 

. a criterion that shows what percentage of review questions he 
wants to pass before leaving the unit 

a Goal Vector that shows the sequence generator the list of 
maps he is scheduled to see because he missed review questions 
based on them 

. a User Display List that shows what blocks from each map are 
to be routinely displayed to him. 



c^^^ued on next page 






THE REVIEWER MODE, continued 



(continued) 

The 

Guided 

Review 



If the supervisor’s switch settings permit it, the reviewer can 
change his display list, ask for other blocks,, go forward or 
backward in the course materials. He may ask for feedback ques- 
tions. and they may be evaluated but they do not influence the 
sequencing of displays — only review questions affect that. 



In short, the generator in the guided review condition finds out 
what the person does not know and then takes him through relearn- 
ing sequences that are tailored for his particular objectives 
and personal background. 



Free 

Review 

Condition 



IMrestricted reviewers can go through the materials more or less 
as they wish. They may be as free as the top level initial 
learner or the referencer. All commands are available to them. 

Many different strategies can be used. For exanple, reviewers 
can set up in advance the list of maps they want to see. This 
can be done by selecting from the table of contents or it can 
be based on the results of optional pretests. Maps indicated 
by the reviev/er would be set into his Goal Vector and then the 
generator would go throu.^ the series with him. He can use 
feedback questions or not, as he wishes. In any case, they 
would not affect his sequencing plan. 

Another reviewer might want to loo^^ over only the general maps 
in a course — the overviews, previews, summaries, conpare-and- 
contrast tables and the' like. These could be set into his 
Goal Vector to save having to ask for each one individually 
in the session. 



Some may like to review by testing themselves on a feedback 
question from each map. Only when they miss or feel uncertain 
would they call for information blocks from the related map. 

pie reviewer whose switch settings allow him conplete freedom 
is even free to ask for a guided review. 

In short, the free review condition helps the reviewer follow 
any strategy he prefers. If his skill in manipulating the 
system is weak, the generator can be set to teach him that also. 



liO 
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THE REFERENCER MODE 



Introduction 



Brief 

Description 



The 

Inexperienced 

User 



The Learning-Reference System provides a number of features to 
facilitate the retrieval of information from the course data base. 
The person who comes to the system with a reference problem is 
served by the Referencer Mode whose primary objective is to 
provide quick location of those concepts, procedures and so forth 
that are relevant to the user’s question or problem. 



Most often referencers approach the system with specific questions 
in mind and in many cases speed of retrieval is important. To 
accoii5)lish his search, the system allows the referencer the 
greatest amount of freedom in selecting the materials he wants 
to see. When the user logs on and the Mode Switch is set to 
Referencer, then automatically all relevant switches are set at 
the free end of the control continuum and all commands are legal 
for the referencer to use. 



Basically the referencer does not have to know very much about 
the system in order to retrieve the information that he is 
seeking. When his interaction with the system begins, the 
generator asks him to select a course and then shows this display: 



TO FIND WHAT YOU WANT* YOU GIVE COnHANDS BY PUSHING 
COnriAND BUTTONS AND BY TYPINGi 



• TO SEE THE INDEX. PUSH THE INDEX BUTTON. 

• TO SEE THE TABLZ OF CONTENTS. PUSH THE BUTTON UITH THAT 
LABELi 

• TO SELECT INFORTIATION YOU WANT TO SEE. PUSH THE GOTO 
BUTTON AND THEN TYPE THE NAME OR NUMBER OF THE MAP YOU 
UANT. 

• TO CHANGE COURSES. PUSH THE CHANGE COURSE BUTTON. 



TO FIND OUT UHAT OTHER COMMANDS DO. PUSH THE "DESCRIBE" 

BUTTON AND THEN PUSH THE BUTTON WHOSE FUNCTION YOU WANT TO 
KNOW ABOUT. 



If the user inserts an inappropriate command, the system will 
prompt him by suggesting commands that might be more useful. 



When the referencer uses the GOTO command, the generator shows 
all blocks from that map except feedback questions. If the 
user wants to see those he. must use the FEEDBACK command. 



continued on next page 
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THE REFERENCER MODE, continued 



The 

Experienced 

User 



Future 

Plans 



The person who is experienced in operating the system can use 
some additional techniques to save himself time. 

If, for instance, he knows that he has a series of things to 
look up, he can set up his own Goal Vector so that the maps he 
wants to see are tagged for display. Otherwise, he would have 
to consult the table of contents and use the GOTO command to 
bring up each map in the series. 

If the referencer wants to check his understanding of a map 
concept, he can ask for a feedback question to be displayed. 

He does not have to answer it, but he may if he chooses and he 
may also ask for evaluation of his- answer by pushing the JUDGE 
command. Kis answer, whether right or wrong, has no effect on 
the sequencing of displays because the referencer has no require- 
ments he must meet. 



Few concrete data are available concerning the behavior of -those 
who are doing reference work. We expect that by debriefing 
interviews and by keeping a record of certain data on -the 
commands used by those in "the Referencer Mode, we may learn of 
modifications -that may serve this user class more efficiently. 
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FOOTNOTE ON THE BROWSING MODE 



Introduction Originally project objectives included a provision for designing 
services that would enable a person to browse throu^ the infor- 
mation base of a coirputer system in the way that he would 
browse in a library or bookstore. 



Browsing 

Behavior 



Future 

Plans 



Patterns of browsing behavior have never been very well documen- 
ted, althou^ J, C. R, Licklider has suggested some experimental 
studies of browsing in order that automated libraries will not 
destroy this rewarding pursuit ( Intrex, ed. by Overhage, 

C.F., and Harman, R.J., M.I.T. Press, 1965). 

When we considered the kinds of things w^ ourselves did when 
browsing among books, we noted sudi things as being attracted 
to a book because of its arresting cover design, the presence 
of either a favorite topic or of a strange, puzzling term in 
its title, and so on. 

At the present time we see no services needed by the browser 
that are not now provided by the Referencer Mode. The initial 
single display describing the four or five needed commands and 
then the free control settings should be quite sufficient to 
let the browser explore the information base at will. He 
would need access to all courses and it may be found that for 
him additional system-wide indexes and tables of contents are 
required. 



For research purposes, it might prove valuable to keep separate 
records of browsing and reference behaviors. That can easily 
be done by having the user log in and be counted as a browser, 
and then program control would be shifted to referencer 
decision tables. Iftitil additional needs of browsers have been 
identified, we plan to give no further attention to this mode. 
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THE BRIEFING MODE 



Introduction 



Description 



Formal 

Briefing 



The 

Iftirestricted 

User 



One of the primary purposes of the Briefing Mode is to acquaint 
users with the general nature and content of aiiy course in the 
system. The generator can provide special materials and sarrole 
lessons to enable the user to: 

. learn generally what the course is about 
. conpare one course with another 

. decide whether he wishes to take the course in the role of 
either learner or reviewer. 



In order to show the general nature of a course, the generator 
can present information about : 

. the difficulty level of the course 
. names of prerequisite courses 
. number of maps in the course 

. the time estimated for conpleting the course as a learner 
. course objectives 

. the main points or big ideas of the course 
. overviews and previews of some units 
. saiple lessons and sample tests. 

These kinds of information are stored in every course data base, 
^ Chapter 3 mentioned. The author of each course is responsible 
for providing the details and also for tagging other maps that 
are suitable for briefing purposes. 



Since some who mi^t want to use this mode may not know, how to 
operate the system and use the commands, the system offers a 
formal briefing in which the sequence generator takes charge of 
selecting displays. The client may indicate how much time he 
wants to spend and the presentation will be adjusted accordingly. 

Only a few commands are available for the person who receives 
the formal briefing— he can of course use NEXT to move the display 
along and he can ask for the table of contents and index. He can 
record comments, ask for help, change courses and stop. Other- 

the displays are totally determined by the sequence generator. 

The ^restricted user is free to explore the course material as 
he wishes. The system can suggest certain types of overview and 
general materials that are appropriate and it offers certain 
aids such as the "course comparison table” that permits him to 
check the dimensions along which he would like to compare two 
courses. For example, he mi^t like to see how two courses com- 
pare on the items listed in the Description block above. The 
comparison table can be displayed to him or can be printed for 
him to examine off-line. 






OTHER USER CLASSES 



Introduction Although the design o£ modes for support personnel was outside 
the scope o£ the present project, it was natural that we should 
begin to see something of their outlines as we filled in the 
picture for the primary clients. Some support functions were 
occasionally worked out in detail, others were sketched in only in 
a tentative way. 

Just how these other modes will cater to the individual needs 
of the users is not fully clear yet, but the techniques can be 
e:q)ected to parallel those of the other modes. In addition, 
special aids will be made available for some functions. 

We report the general status of these modes here. 



Supervisor Because of the in^ortant role played ten5)orarily by the super- 
sede visor in this first version, a number of functions of the 

supervisor have been in^lemented throu^ the decision-table 
level. This degree of detail was useful in conceptualizing 
how future versions could automatically accon^lish these same 
tasks. 

The major function of the supervisor is to provide the sequence 
generator with all the information it needs in order to serve 
a newcomer to the system. 

While it is easy to conceive of a set of professional require- 
ments desirable in a training supervisor, the specific tasks 
assigned teii5>orarily to this siq>ervisor require little educa- 
tional e:q)ertise and only superficial acquaintance with this 
system itself. In other words, we are concerned at the moment 
with the minimum that a supervisor must do to set the system 
displays going for a user. 

The system has the capability at present to deal with a 
hierarchy of supervisors in which the operations of more exper- 
^ fenced supervisors are less restricted. Nevertheless, the 
main work of any supervisor is accomplished throu^ an on-line 
interview in which the system asks for the information it 
needs. 

It asks for specific data, pronpts the supervisor on unset 
switches, and displays menus from which he can select the 
alternative appropriate for the given user. Thus little burden 
is put on the supervisor’s memory or training background. 



continued on next page 
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OTHER USER CLASSES, continued 



Writing, 

Editing 

Modes 



Researcher 

Mode 



Several classes of support personnel may be needed to prepare 
course materials for the system: authors, instructional 
designers, editors, key-punch operators and programmers. Space 
has been tentatively allotted for separate modes for each of 
these user classes, but it is most likely that the functions 
will be combined into one or two modes that can serve them all. 

Several existing languages have been considered with regard to 
their suitability for authors and instructional designers in 
this system, and some sets of requirements have been con5)iled; 
however ^ in-depth study leading to solution of this issue was 
not within the purview of this project. 



The author aids that have been designed so far include a demon- 
stration set of questionnaires to guide authors in giving the 
generator every last detail it needs about each information 
block. The data required for even one feedback question is 
tedious to such a mimbing degree that only a structured form 
and preferably an automated procedure can prevent careless 
omissions, 

(Each feedback question is accompanied by such. details as these: 
its associated map, its number, name of block it is based on, 
its difficulty rating, the number of different answers expected, 
the different answers themselves to be matched against the 
user’s response, the correct answer, actions to be taken for 
the different possible answers, display messages geared to the 
response alternatives, the number of times the user can try 
answering, and various flag and switch settings to indicate 
sequencing or display instructions.) 

For all personnel engaged in preparing course materials fromi 

the instructional designer to the key-punch operator— the 
intention is to streamline their tasks by making procedures 
explicit, and by providing special commands and system guidance 
that will enable them to function with a m-iTiirrmm of special 
training on this system. 



The system is built to retain detailed records of each user’s 
interaction with the system— not only his identity, personal 
data, location, date, time and purpose, but also his path 
through the materials, blocks called for, command usage, per- 
formance data, control conditions, display conditions, feedback 
conditions, and so on. 



continued on next page 
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OTHER USER CLASSES, continued 



(continued) 

Researcher 

Mode 



Such data are not only essential for both system and information 
inprovement but they are a gold mine for research purposes. 

They can be almost useless unless the researcher is given the 
tools to tabulate and summarize them in the form he requires. 



Future plans call for programs that enable the researcher to 
extract what he needs with a minimum of special system training. 
Special commands, displays of data- condensing qptions, con5)u- 
tation packages, provisions for off-line processing and print- 
out formatting choices are some of the potential aids that 
could make the Learning-Reference System a first-class research 
faci lity. 
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SUMMARY OF GENERAL INDIVIDUALIZATION STRATEGY 



The 

General 

Strategy 



Whether the user be a prime client or one of the support 
personnel classes^ the general strategy followed in programming 
the generator is the same: to make it respond to each indivi- 
dual by basing its actions in part on: 

. the user’s objectives, capabilities, and wishes 
. his familiarity with the system 

. the current state of his knowledge of the area 
in which he is operating. 

Varying degrees of system guidance can be chosen or assigned. 
Built-in aids — conputer-directed interviews, brief operating 
instruction sequences, questionnaires, and explicit step-by- 
step procedures for some modes — all are designed for the 
purpose of making each user’s experience with the system as 
effective and effortless as possible. 
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CHAPTER 6 CREATING AND MAINTAINING SYSTEM DOCUMENTATION 
OVERVIEW OF THIS CHAPTER 



Introduction The process o£ designing a coii5)lex system always brings with 

it the problems o£ trying to keep descriptive paperwork £rom 
lagging hopelessly behind design solutions. Because o£ 'Che 
nature o£ the Leaming-Re£erence System design project, it 
was particularly iii5)ortant that we have documentation keep 
pace as the design evolved. 

Consequently as a major task in this project we made a concerted 
ef£ort to work out explicit procedures and £ormal communication 
arrangements not only for keeping project documentation parallel 
to design work but also for consistently carrying through modifi- 
cations in the plans. 

The set of procedures, guidelines and special aids that were 
developed to make documentation and t5)dating systematic and 
current is known as the Documentation-Updating System. It 
is considered part of the larger body of data developed about 
Information Mapping and its application. 

This Chapter This chapter outlines briefly the nature of the system and 

describes how documenting and iq>dating procedures are 
organized. 
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NEED FOR DOCUMENTING DURING DESIGN DEVELOPfCNT 



Introduction 



Some 

Relevant 

Factors 



To design a con^uter-based system for learning and reference, 
we realized, would be a complex and tedious process, extending 
over many months. These and a number of other factors in the 
design project made concurrent documentation especially 
in5>ortant. 



Initially certain calculations about the course of the design 
project made us appreciate the need for documentation; 

. Designs of complex systems evolve in cycles of 
successive approximations and even though this 
was to be a first-pass version, we knew it would 
change continuously 

. during this project itself, and 
. perhaps over a several year period. 

. The information to be developed would be coii5)lex, 
full of smr.ll details intricately interrelated and 
therefore outside the range of ready recall. 

. The design would be the work of four or five staff 
members, each with his own area of responsibility, 
who needed to share solutions and plans at a very 
current level if they were not to work at cross 
purposes to one another. 

. The possibility existed that some staff might be 
dispersed at other locations in the country. 

. There was also the probability of some personnel 
turnover, especially among support personnel.' 

. There was the likelihood that there might be a lag 
of several years between design of the system and 
the beginning of its implementation. 

. Those who would carry out the in^jlementation would 
probably not be those who created the system. 



THE NATURE OF THE DOCUMENTATION-UPDATING SYSTEM 



Introduction 



General 

Objectives 



The Main 
Features 



Comment 



The development of a set of organized procedures, policies and 
aids for documentation was an integral part of the present 
project and early in the first chapter it was mentioned as the 
fourth major task. 

Its rationale and key ideas were worked out even before the 
design project got under way; its procedures and policies were 
developed, tried out on the design project and in^jroved 
through try- and- revise cycles. 



Our objective was to develop a system that would encourage 
the production of documentation at such a level of detail and 
in such an accessible and imderstandable form that it would 
serve: 



. the creators of a system during the design process 

. designers who are responsible for adapting a system 
design for in^lementation in a given facility 

. programmers who must translate a design into 
operational programs 

those who eventually assume responsibility for 
operating the system 

. system managers who in later years must consider the 

feasibility of modifying or adding major system functions 
or of adapting existing functions to new equipment. 

The central ideas of the system are that: 

. Documents are written and organized in Information- 
Mapped form. 

. They are maintained and updated according to explicit 
standardized procedures and policies. 

. The personnel who perform the work communicate with 
project staff through special record forms that serve 
as e^qjlicit information- channel ling instructions. 

These topics will be considered separately. 



The system that we describe is still evolving — it improves and 
expands as our experience with different situations grows . 
Ultimately, it is expected to become an automated system for 
document maintenance . 
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DOCUMENTATION IN IN FORMAT I ON -MAP FORM 



Introduction The key idea of the system is that Information Mapping 

is the basis for all document preparation and organization. 

This means that documents are kept systematically in a form 
that facilitates scanning and retrieval. And, because maps 
are movable modules, documents can easily be expanded without 
upsetting the organizational scheme. 

Writing The information that forms the backbone of project documents 

Documents must come from the creators of the system, and yet these are 

often the people from whom it is hardest to extract systematic 
information. Commonly we find that: 

. The thinking of creative people runs far ahead of 
their written output. 

. They find the task of writing down their ideas in 
detail intolerably burdensome. 

. They do frequently write themselves cryptic scribbled 
notes about an idea or possible future plan. 

Information Mapping applied to the documentation task will not 
solve the problem of extracting information from designers, 
but it has developed procedures that relieve them of some of 
the more burdensome aspects of the job. It even finds a place 
for their scribbled notes to themselves so that the system 
does not risk losing an important idea. 

Whether system designers can do their own documenting or must 
work with technical writers who can elaborate their notes or 
conversations into formal documents, the aids that Information 
Mapping can offer are these: 

. explicit procedures that enable support personnel to 
take over the burdensome aspects of document handling 

. a set of guidelines and procedures for writing documents 
in standardized, organized form. 

The kinds of procedures that are carried out to keep documents 
properly ordered, organized, and updated are described in the 
following map. 

For the writers of documents. Information Mapping provides a 
ready-made set of instructions and guidelines. These were 
described briefly in Chapter 2 and are given in detail in the 
^J^ual, A Reference Collection of Rales and. Guidelines f or- 
Writing intormation -Mapped Materials (cited in Chapter 1*^^ 

•wontinued on next page 
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DOCUMENTATIOiM IN IN FORMAT I ON -MAP FORM, continued 



Use of 

Decision 

Tables 



In connection with both the design and documentation of 
con5)uter software, we find decision tables especially useful 
in that, compared with flow charts, they are: 

. faster to construct 

. more efficient in format for verifying logic 
. easier to revise. 



Since we chose to build the sequence generator on decision 
\ tables because of their flexibility, our system contains 

\ many procedures that enable tables to be constructed and 

revised by support personnel who need have no understanding 
\ of the system. 



Physical The organization scheme for project documents will follow the 

Organization nature of the project and must be worked out on an individual 
of Documents basis, although commonly it proceeds from general descriptive 

material about purposes, specifications and major conponents 
toward increasingly fine detail concerning each aspect. 

In our own application of the system, documentation is kept 
in loose-leaf books because of the frequency with which maps 
are added or modified. Each member of the project staff has 
his own copy of the documents. 
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STAI4DARDIZED MAINTENANCE AND UPDATING PROCEDURES 



Introduction 



Description 



The Next 
Maps 



Maintenance and updating tasks are sin 5 )le and utterly tedious. 
Their successful accon^lishment depends upon making detailed 
step-by-step instructions covering what must be done for each 
situation. 

The set of procedures and policies that we evolved are carried 
out by support personnel manipulating reams of paper, but 
clearly the same procedures could be carried out on-line, 
eliminating some of the paper and most of the tediousness of 
the human tasks. 



The full accoimt of the existing procedures and special aids 
appears in Section 800 of the Project Document cited. Chapter 1. 

In general the procedures, rules and instructions described 
there consist of: 

. instructions for adding, deleting, and modi^ing 
numerous aspects of the documents — maps, tables, 
sections, terms, and so on 

. instructions for carrying changes through to all areas 
affected by the modification 

. special aids such as tables, lists, questionnaires, 
record forms and so forth that are intended to stream- 
line staffwork in applying the procedures 

. formatting and typing policies made explicit for typists 
and editors 

. various features to keep system designers informed 
of dianges. 

While the contents of many of these procedures are specific to 
the individual project, the principles and general policies 
are readily adaptable to other situations. 

Rather than attempt to describe each of the above classes of 
materials, we give a series of examples that illustrate some- 
thing of the nature of the system, (.'ihe actual procedures 
are given in the Project Document mentiovied above.) 
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DOCUMENTATION SYSTEM EXAMPLE: PARTIAL TABLE OF CONTENTS, UPDATING SYSTEM 



Introduction This san 5 )le from the Table of Contents of the Updating Procedures 

Section is reproduced here to show the kinds of tasks that must 
be made explicit. 



Section 800 — The Updating System, continued 

. How to Index a' Section of the Document -..63250 

. How to Index Terms 63500 

. How to Use the Index to .Change the Name 

of a Term 63600 

. How to Add Terms to the Index 64000 

. How to Delete Terms from the Index 64300 

. How to Combine Synonymous Terms in the. Index 64600 

. Types of Terms to Index in Any Document 65000 

Terms to Index for This Project 66000 

. How to Index Variables, Flags and Switches 66200 

. How to Index a Command 66300 

. How to Index Decision Tables, 66500 

. How to Update the Index from a Changed Map 66600 

. How to Update the Index from a Changed Decision 

Table 66800 

. The Index for Publication 67000 

. When to Prepare an Index for Publication 67300 

. How to Prepare the Index for Publication.. 67600 

Updating Specific Maps in the Document 68000 

. How to Update the (Where Each Flag is Mentioned) 

Map .' 68200 

. How to Update the (Where Each Switch is Referred 

to in the Tables) Map .....68400 

. How to Update the (Which Tables Mention Other 

Tables) Map... 68600 

. How to l^date the (Where Each Table is Referred 

to in Other Tables) Map 68800 

.. Adding Specific Types of Terms 70000 

. How to Add a Flag 71000 

. How to Add a Variable 72000 

. How to Add a Switch 73000 

. How to Add a Command 74000 

. How to Add a Table 75000 

. How to Add an Attribute 76000 

The l^dating Packet of Maps 81000 

. When to Update All Outstanding Copies of the 
Document 82000 



End of examgZe 



DOCUMENTATION SYSTEM EXAMPLE: STARTING POINTS FOR UPDATING ORDERS 



Introduction Orders to change documentation come to an updating assistant. 

The table below shows him where to start in processing such 
orders . 

The nature of the change is located in the first two columns, 
the appropriate starting map for each is shown in the righthand 
column. These latter maps contain decision points that lead to 
other procedural maps within the system, thus insuring that the 
entire document is properly iq>dated. 



ORDER 


ITEM REVISED 


WHERE TO START 


Index 


, any term 
. any map 


HOW TO INDEX TERMS, 800/u5500 




Section of the 
document which does 
not contain 
decision tables 


HOW TO INDEX A SECTION OF THE DOCUMENT, 
800/63250 




Revised map 


HOW TO UPDATE THE INDEX FROM A CHANGED 
MAP, 800/66600 




Revised Decision 
Table 


HOW TO UPDATE THE INDEX FROM A CHANGED 
DECISION TABLE, 800/66800 


Change 1 


1 . name of a tern 


HOW TO USE THE INDEX TO CHANGE THE NAME 
OF A TERM, 800/63600 


Add 


Map 


HOW TO ADD A MAP, 800/56000 




Section 


HOW TO INSERT A NEW SECTION, 800/43000 




Flag 


HOW TO ADD A FLAG, 800/71000 




Variable 


HOW TO ADD A VARIABLE, 800/72000 




Switch 


HOW TO ADD' A SWITCH, 800/73000 




Command 


HOW TO. ADD A COMMAND, 800/74000 




Decision Trble 


HOW TO ADD A DECISION TABLE, 800/75000 



continued on next page. 
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DOCUMENTATION SYSTEM EXAMPLE: STARTING POINT FOR UPDATING ORDERS, continued 



ORDER 


ITEM REVISED 


WHERE TO START 


Add 

(cont) 


Attribute 

Term 


HOW TO ADD AN ATTRIBUTE, 800/76000 
HOW TO ADD TERMS TO THE INDEX, 800/16000 


Date 

and/or 

Number 


• any item 


HOW TO DATE A MAP, 800/32000 
HOW TO NUMBER A MAP, 800/51000 


Update 


WHICH TABLES 
MENTION OTHER 
TABLES map 


HOW TO UPDATE THE "WHICH TABLES MENTION 
OTHER TABLES" MAP, 800/68600 




WHERE EACH TABLE IS 
REFERRED TO IN THE 
OTHER TABLES map 


HOW TO UPDATE THE "WHERE EACH TABLE IS 
REFERRED TO IN THE OTHER TABLES" MAP, 
800/68800 




WHERE EACH FLAG IS 
MENTIONED map 


HOW TO UPDATE THE "WHERE EACH FLAG IS MEN- 
TIONED" MAP, 800/68200 




WHERE EACH SWITCH 
IS REFERRED TO IN 
THE TABLES map 


HOW TO UPDATE THE "WHERE EACH SWITCH IS 
REFERRED TO IN THE TABLES" MAP, 800/68400 




All copies of the 
document 


THE "WHAT IS NEW" FUNCTION, 800/83800 




Table of Contents 


HOW TO MODIFY THE TABLE OF CONTENTS, 
800/24000 


Delete 


Section 


HOW TO DELETE A SECTION, 800/42000 




Map 


HOW TO DELETE A MAP, 800/57000 




Terms 


HOW TO DELETE TERMS FROM THE INDEX, 
800/64300 



12 ? 
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DOCUMENTATION SYSTEM EXAMPLE: MISCELLANEOUS AIDS 



Status 

Remark 

Sheet 



Request for 

Clarification 

Sheet 



The Status Remark Sheet record form provides a means for a 
system designer to communicate about changes he wants madej 
future plans, points of disagreement on some aspect of the 
system. It may be addressed to specific staff members or to 
the project director, but in any case it will go through a 
central clearing point. 



The Request for Clarification Sheet is a standard fonn by means 
of which questions on some aspect of the system may be routed 
to the appropriate person. It may be used by all project per- 
sonnel and it is always routed through the updating assistant 
who keeps track of pending questions . 



"What - s 
Service 



All principal staff members have copies of all project docu- 
ments which are kept up-to-date by support personnel. All 
receive lists of changes that have been entered, but there 
remains the problem of calling their attention to significant 

changes. Special periodical summaries are issued to take care 
of this. 



Short 

Courses 



Selections from the documentation of various policies, procedures 
and formats are easily assembled into self-instructional courses 
for typists, clerks, writers, prograjonners and the like. By this 
means we have found it economically feasible to employ ten^orary 
personnel. The existence of such ej^licit base materials also 
has minimized project slowdowns due to personnel turnover.. 
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SKETCH OF DAILY DOCUMENTATION ACTIVITIES DURING SYSTEM DESIGN 





Daily Changes in 
Data Base to 
Everybody's Copy 



DOCUMENTATION 



Introdviction 



Management 

Support 



Priorities 

and 

Responsi- 

bilities 



POLICIES 



Any program for documentation will work only so long as it has: 
, top management support 

. clearly drawn lines of decision-making authority and of 
responsibility for approving document entries 

. an established list of priorities for accomplishing 
the various classes of tasks. 



If documentation is to be accurate and complete on the one 
hand,, and useful to the community of creators on the other, 
it must begin when the design phase begins. 

Too often documentation is the final mopping-up task after 
the project has been finished, ideas have groTO cold, and staff 
has grown Impatient to turn to other things, Documents could 
hardly fail to be superficial and incomplete* 

To get good documentation, top management must support it 
seriously with a top priority rating and sufficient funding. 
TTiey must also lecognize its role as an important tool through- 
out the actual design process. 

If documents are begun at once, made easily accessible, and 
kept current, designers will derive much help from them. In 
complex projects a designer cannot keep in mind all the 
consequences of a modification he makes even in his own area, 
let alone the effects it might have on the work of 
others. But system documents with lists that track each idea 
to all references in the system let a designer see ramifica- 
tions of a change that ho could not otherwise appreciate. 

Thus documentation must be perceived by management not as 
constituting the final straw at the end of a long project but 
as playing a vital role in the design process from the 
beginning. 



Documentation can become a morass unless explicit policies are 
formulated to control the movement of materials in and out. 

TTie statement of such policies, lines of authority and lists 
of priorities should be worked out by project managers before 
documentation begins. 

Periodic reviews should be slated to examine the efficiency 
of the procedures and to revise them when necessary. 
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Emphases Tlie Documentation-Updating System is based upon these 

convictions i 

• documentation should begin when design v/ork begins 

• document ation I pToperly executed j is an important 
tool during the design process 

. change must be taken for granted and planned for 

• docuinenting and updating procedures can be made so 
explicit that consistent actions are taken and 
changes followed through 

, functions involved in documenting can be effectively 
separated and assigned to specialists in such a way 
that system creators are spared the more burdensome 
aspects 

• documentation needs Significant management support. 

Mapping is well suited as the basis for documenta- 
tion because: 

• its formatting ^d reference features facilitate 
scanning and information retrieval 

• its modular form permits documents to be expan, ded ^d 
modified without upsetting the organizational structure, 

me System The Documentation-Updating System consists of: 

, a set of procedures for writing and organizing 
documents in Information Mapped form 

. a set of standardized prooedures and policies for 
maintaining and updating the documents. 
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CHAPTER 7 SUMMARY OF THE LEARNING- REFERENCE SYSTEM 
OBJECTIVES OF THE SYSTEM 



Chapter 

Preview 



Primary 

Objectives 



Previous chapters have sketched in the main features of the 
Learning-Reference System. Since we have talked of many things 
the system can do and of the many mechanisms it uses, we pro- 
pose in this chapter to 'iraw it all together in a quick 
recapitulation of what the system does. 

This summary may be a convenient reference aid in later 
discussions of assessment and future plans. 



TTie primary objectives of this system are to provide computer- 

based information services for: 

■ initial learners : the service to consist of sequences of 

information displays that are either chosen by the learner 
himself or that are re commended for h im by the system 
after evaluating: 

. his goal in learning and the level of mastery he seeks 
. his background experiences, abilities, and attitudes 
. his behavior and preferences shown during learning sessions 

. reviewers : the service to consist of the following aids to 

the person who wants to refresh his memory: 

. evaluation of his knowledge of the area 
. an individualized, guided review of those materials that 
he has forgotten or wishes to see 
. sequences for relearning either under system control or 
under the user's own direction. 

• reference users: the service to consist of aids to enable 

the user to idehtlfy quickly the information he needs and to 
call it up for display, 

. briefings : the service to consist of displays of information 

about any specified course, giving the client an overview of 
the course, its aims and requirements, 

. browsing ; the service to consist of aids to enable browsers 
to move about in any manner they like. 



continued on next page 



OBJECTIVES OF THE SYSTEM, continued 



Secondary 
Objectives 

. to prepare inforniation materials 

. to collect performance data on system functions and on 
information units for the purpose of guiding continued 
improvement of the system 
. to update information units 
. to modify system functioning 

. to collect for research purposes various data about system 
functions, information usage and clients* behavior patterns. 



The secondary objectives are to build into the system those 
StUpport services that will make it easy: 



MAIN SYSTEM COMPONENTS AND THE FUNCTIONS THEY ACCOMPLISH 



1 

Modular 

Information 

Base 



2 

Command 

Language 




Sequence 

Generator 



The contents of the information base may be characterized 

as follows: 

. written subject-matter materials are classified by Information 
Mapping categories and are stored in labelled Information 
blocks that can be individually called up and displayed as 
required by each client 

u complete descriptive details auout the infonnation blocks 
and their various groupings into maps, units, and courses 
are stored in the data base in order to permit selection of 
materials suitable for the individual client 

• data about the clients are maintained in order to make 

possible delivery of infonnation sequences that are respon- 
sive to their needs, 'Ilie data may include: 

. purpose of user and level of mastery sought 
. previous experiences, abilities, attitudes 
. performance on prerequisite tests and pretests 
. switch settings that deteMiine how the client and 
sequence generator will interact 
. various flags, lists, etc., that keep track of the flow 
of action during each client session so that meaningful 
sequences are displayed. 

. data about system c om ponents are stored to enable the 
sequence generator to accomplish various system functions 
such as reacting to commands . 



TTie command language consists of a set of terms designed to 
enable the user to direct the computer and to respond to 
displays. The set is capable of being restricted for some 
classes of users. 



The sequence generator is a computer program based upon a set of 
decision tables that handle all transactions between users and 
the information base. It not only fetches information for 
display but also keeps records of the interactions . The 
generator has the capability of responding in the ways described 
below, but not all clients will be allowed to take ad ant age 
of them . 
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Ccontinued) 

3 

The 

Sequence 

Generator 



Tlie generator operaLt-'i in different modes dcpniding on the 
user's purpose: 

. Initial Learner Mode: 



. for each learner a learning goal is established in terms 
of how much of the subject matter he wants to cover 

. for each learner a criterio n is established also to indi- 
cate how well the learner wants to know the materials 

. control of information selection and arrangement can be 
allocated to user or to system or it may be distributed 
between them in various patteriis 

. level of control can be designated either by the user 
himself or by a supervisor, depending on how the computer 
facility is managed 

. the leai'ner with freedom to control the system can use 
all comiiiands and learner services of the system^ he can 
take charge of his own learning path or he can direct 
the generator to take some control over the sequencing 
just as it would for learners with less freedom in the 
system 

. when the generator takes part in sequencing decisions, it 
takes into account many aspects of stored data about the 
learner 

. sequencing also depends partly on the learner's responses 
to feedback questions and to unit pretests 

. feedback to learner may be given or withheld at the 
discretion of the learner himself or of a training 
supervisor 

. the selection of feedback questions or worked examples 
for display may be based on their ratings in terms of 
difficulty level 

. remedial sequences may be presented when a specified 
failure rate is reached on feedback questions 

, a learner may change the standard order in which blocks 
are displayed to him and he may add other blocks to the 
standard pattern 

. the learner may ask to see additional blocks about a topic 
or to try extra feedback questions 
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COMPONENTS AND THE FUNCTIONS THEY ACCOMPLISH, continued 



. the learner may depart from a sequence to revisit a 
previous map and may then return to his place 

* the learner may select the next topic from a list of 
appropriate maps 

. the learner may ask the generator to explain the meaning 
of conunands or to suggest appropriate commands he mlcht 
use next 

. the learner may enter comments about the materials, some 
aspect of the system, or his reactions to it; later these 
are reviewed by the supervisor or course designer. 



. Reviewer Mode : 



. control of information sequencing can be allocated to the 
reviewer, to the generatOT, or to both in a more or less 
equal distribution 

. in the system-dominant condition sequencing depends upon 
the user's responses to review questions at the unit level 

. units that are passed at a specified level drop out of 
the guided review sequences 

. users in the guided review condition require little 
knowledge of the command language 

. reviewers in the freer control conditions may take pretests 
in order to bypass units whose material they already know 

. reviewers in the freer conditions may review the course 
materials as they wish, including calling on the generator 
to take over, routine sequencing tasks 

. reviewers may ask for feedback ouestlons and may have their 

answers judged, but these do not affect the sequencing 
decisions . * 



. Referencer Mode ; 

. the user is allowed complete freedom to use the system's 
facilities as he sees fit 

. four simple conmands enable the untrained user to find 
what he wants; Table of Contents command. Index conmand. 
Goto command, and Change Course command 



continued on next page ^ 
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MAIN SYSTEM COMPONENTS AND THE FUNCTIONS THEY ACCOMPLISH, continued 



(continued) . system will instruct the user if inappropriate conmiands 

3 are used or if help is requested 

The 

Sequence . the user may respond to feedback questions if he likes 

Generator and may ask to have his answers evaluated. 



Briefing Mode * 

. special materials give the client an overview of the content 
and structure of any specified course; the displays, under 
the control of either the system or the user, include: 

. course objectives 
. main points of the course 
. difficulty levels and prerequisites 
. applications 
. sample lessons 

. number of maps and estimates of time to learn. 

. those who come to the system for information about the 
general nature of course materials select whatever kinds 
of information they want or they can ask the system to 
present briefing materials to them 

. those using the briefing mode can set a limit to the 
amount of time they wish to spend and as a consequence 
the generator may reduce coverage of less important 
aspects 

. the mode permits comparison of two courses across 
significant dimensions. 



. Browsing Mode : 

. the design of this mode is not yet complete, but its aim 
Is to help the browser explore as he wishes; it will 
probably be built upon many of the mechanisms of the 
Referencer Mode except that *che browser's freedom will 
extend across courses. 

Supplementary 
Services of 
the System 



. system documentation in Information Mapped form giving easy 
access to all aspects of the system 

, written manual of step-by-step procedures for preparing system 
documentation and for maintaining it in up-to-date condition 



ITie system also supplies support services, including: 

. compaete writing guide for the preparation of Information 

Mapped materials, including questionnaires to guide authors in 
recording descriptive data required by the sequence generator 
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MAIM SYSTEM COMPONENTS AND THE FUNCTIONS THEY ACCOMPLISH, continued 



a ’’What is New” service to notify all system personnol of 
recent changes in the system and to call attention to 
significant differences 

. a computer-directed interview with the training supervisor 
to elicit from him the information that defines how much 
guidance the generator is to give each client 

. procedures for collecting statistics across courses and 
across users 

. printouts of data summaries to aid researchers and training 
supervisors 

. procedures and standard forms by whi ch system personnel such 
as designers, editors, or programmers commimicate about the 
changes they want to make 

. special sets of tables to aid designers, prograiimiers , and 
updating clerks by providing lists of system components and 
of the places where each of them is mentioned in system 
documents 



(continued) 
Supplementary 
Services of 
the System 



short training courses for support personnel. 



Comment 



Certain plans for possible extensions and modifications of 
these functions in a later \ arslon are mentioned in the 
closing chapter. 
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WHY EVALUATE 
Introduction 



Purpose 



CHAPTER 8 EVALUATION OF THE SYSTEM DESIGN 



At various stages in a project it is customary to make an evalu- 
ation of one sort or another. In what sense could we evaluate 
the design for a system? We remind ourselves that evaluation is 
essentially a judgmental process and that it is by no means synony- 
mous with statistical assessment of quantitative data. 

Whether quantitative data are appropriate for the judging 
process depends entirely on the nature of the situation and the 
purpose for which evaluation is to be made. 

The evaluation of a paper plan differs considerably from the 
evaluation of an operating system or of an educational program. 

In the latter cases, performance data, user surveys and the like 
can add an agreeable air of objectivity to the decision-making 
process. With a paper plan, we rave to muster other kinds of 
information to aid in judgment. 

Let us consider first why we would want to attach a value to a 
plan--that is, what would be the purpose of evaluation at this 
stage, nien we can see what kinds of information would be useful 
for the task. 



The puipose of evaluation at the design stage is to reach a de- 
cision about the project’s future --whether to build the system 
essentially as proposed, cancel the project, or revise the plan 
and then reconsider its future. 

Such a decision involves considering these two classes of questions * 

li 111030 concerned with its functions and the mechanisms for ac- 
complishing them: Does it do the kinds of things we want? Is 
there some better way of doing the various functions? Are 
there places where we would prefer some other alternatives? 

2c Hiose concerned with its costs: What will it cost to Implement 
at several different levels from deluxe, say, to a stripped- 
down w'^ersion? How much will it cost to maintain it, make new 
course materials for it, train personnel for it, and so on? 



The problem of assembling information that bears on these questions 
will be considered on following maps. 



Comment 



THE QUESTION OF OFF-LINE SIMULATION 



Introduct:* on 



Possible 
Results From 
Simulation 



Since we can get no real performance data from the system, we 
considerdd the possibility of using the simulation approach to 
try out aspects of systeni functioning. 

Commonly when a real system is not yet in existence, some form 
of physical or symbolic model is constructed to simulate aspects 
of the system design in such a way that they can be tested. The 
piirpose of course is to gather information that will help in 
finally implementing the system. 

Models vary greatly--some are mathematical formulations subjected 
to computer manipulation, others are descriptive models subjected 
to logical analyses, and still others are actual physical proto- 
types that are subjected to real-time use by real people. 

In the earliest stage of the design process, we had conjectured 
that useful data might be obtainable by asking human operators to 
mimic the actions of the sequence generator and thus select 
paper displays of information blocks; the sequences might even 
be used by tryout subjects for learning or for reference work. 

In actual fact, however, the sequence generator evolved into a 
much more versatile and complex program than we had anticipated. 

It was quite cleur that iti, functions wei*e dependent on too many 
details to be simulated by hand. In salecting information se- 
quences, the generator has to check the status of so many com- 
puter locations and to consult and/or modify so many flags, 
switches, lists, stacks and so on that a hand-operated simulation, 
could have no hope of producing displays with any useful frequency 
Real subjects would perish from boredom between displays; thus 
no relevant information could emerge from such tiyouts. 

As a matter of fact a team of staff members did ej^eriment with 
stepping through the programs and decision tables of the sequence 
generator to assemble paper "displays.'' The objective was to see 
if we could speed up our hand-operated simulation enough to war- 
rant data collection with real subjects. Even with practice and 
shortcuts, the process remained unrealistically slow; we found 
that those parts of the sequence generator that we were using did 
"work" in the sense chat they led to the delivery of meaningful 
sequences , . but we discarded the hope of trying real learners or 
reference workers with it. 

Without real test subjects, the only conceivable utility of con- 
tinuing such an exercise would have been to check the linkage 
between progran elements; that however is a trivial task better 
done by the routine desk- checking procedures of the computer- 
prograimning field. 
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THE QUESTION OF OFF-LINE SIMULATION, continued 



(continued) 
Possible 
Results From 
Simulation 



Apart from the question of system functioning there is the 
question of whether a simulation of some sort could test the 
adequacy of the Information Mapped data base--that is, whether 
the different possible arrangements of blocks drawn from the 
information base make sense and are acceptable to the different 
classes of users , 



Conclusion 



We already had a collection of blocks cut from the Information 
Mapped book on Sets and individually mounted on index cards. 

These were useful for testing the adequacy of our guidelines 
for preparing course materials for the proposed system. 

Now would a simulation using this collection of information 
blocks with tryout subjects help us in assessing the meaningfulness 
of reassembled information blocks? We think not--it would tell us 
something about the specific product, the book on Sets, and it 
might detect the need for a change in the writing rules . But 
shortcomings in either case would be corrected in routine devel- 
opmental testing. 



At this stage of the design, it is not possible to nroduce quan- 
titative evidence bearing on its effectiveness. We do, however, 
argue that: 



. the basic conceptions underlying the system have been sub- 
jected to repeated logical analyses by us and their soundness 
may be tested independently by anyone who wants to examine the 
detailed system pj.an. 

. all components of the system, including information base organ- 
ization and sequence generator details, are so designed that 
they can easily be improved according to procedures that have 
already been specified. 

In short , the effectivenass of the proposed system can best be 
judged by everts who study the project documents. And there is 
the added assurance that a specified quality level is achievable 
through the built-in provisions for developmental testing and 
revision. 



Apart from the question of effectiveness, however, other aspects 
of the system must be considered in reaching a decision about its 
future. These aspects are discussed on the next map. 
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BASIS FOR EVALUATION 



Intro ductior 



Services 

\ 



PreDaration 
of Course 
Materials 



o 




^ether a system has value for a given client depends upon v/hether 
it supplies the services he needs at a price he can afford. 

Some of the questions that one would nsk in order to judge the 
Learning-Reference System are: 

, Does it provide the services you want for your clientele? 

V/hat tasks would be required to implement this system? 

. How long would it take? 

. What range of costs might be associated with different versions 
of the system? 

. What is required to operate a system like this? 

Information concerning each of these will be considered. 



Only a prospective client can judge whether the system provides 
the range of services he wants. In the case of the Learning- 
Reference Systemj he can ascertain that fact by comparing his 
requirements with some statement of the characteristics of the 
system. The material in Chapter 7 may be a convenient source; 
if the summary statements seem promising, the prospective client 
would want to follow up by study of the system documents. 



For the Learning-Reference System, the tasks required for pro- 
ducing course materials are much the same as for producing 
mapped books. However, there is also the work of storing extra 
examples and feedback questions, and of recording details about 
the materials for the sequence generator’s use. 

In 1969 (Horn, Nicol, Kleinman, Grace) when the tasks and costs 
of Information Mapping books were estimated, we judged that the 
time spent on the major tasks of production was distributed this 
way: 



. curriculum planning 


25% 


. writing and editing 


55% 


. developmental testing 


20% 



ITie direct labor cost for materials sufficient for an instruc- 
tion hour was estimated to run around $1000. (Organizationai 
overheads are not included and may run up to 150% of direct labor.) 

continued on next page 
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BASIS FOR FVALUATION, continued 



(continued) 
Preparation 
of Course 
Materials 



Training 

Writers 



System 
Design and 
Progranuning 



Materials for the computer'-based system can be expected to cost 
more since the writing tasks are somewhat greater. In view of 
the extra work and rising costs, our best guess now is that the 
materials for an hour of instruction would run in the neighbor- 
hood of $1500 (direct labor cost) . 

Materials for an hour of instruction have been used as the basis 
for the estimates because they cor stitute the major writing chore 
for the information bases of the Learning-Reference System. It 
should be remembered, however, that these materials serve other 
purposes as well. Strictly speaking, then, we should point out 
that the cost estimates are for an hour of materials for instruc- 
tion, reference, review, and briefing. 

The cost of preparing materials, in probably be reduced if the 
work is broken down into separate fuictions and each assigned to 
a specialist. For example, the initial writing can be done by 
the subject-matter expert; the draft can be turned into Informa- 
tion Mapping form by a mapping specialist’ support personnel can 
record the attribute data about the materials in the form required 
by the computer, and so on. We are experimenting with this pro- 
cedure now but have no cost data available for comparison. 

The above figures do not include the costs of training writers . 

Our experience in training for Information Mapping suggests that 
a one-v?eek course followed by several weeks of practical experi- 
ence under a competent editor is sufficient. 

The Leaming-Referfcnce System documentation Includes formal 
questionnaires that quickly draw from writers the exact informa- 
tion about the materials that the system requires. Other proce- 
dures and checklists aid in streMilining the writers' work and 
reduce training time . 



The selection of a computer system will involve consideration of 
many aspects of the clients' needs and resources. Once that de- 
cision is reached, system design work will be required in order 
to adapt the Learning-Reference System to the facilities and 
software of the chosen computer. 

This involves the working out of file management procedures, data 
base formats, data collection procedures and the final decision 
concerning what sequence generator details are to be implemented. 
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BASIS FOR EVALUATION, continued 



Ccontinued) 
System 
Design and 
Programming 



Time and 
Costs 



Programming, coding, and debugging the sequence generator come 
next. Course materials must be translated from paper to the 
computer data base. And finally a period of system integration, 
checkout and revision will be needed. 



The amount of work involved in each of these tasks will vary with 
the sophistication of the system and its ready-made facilities. 

The time and cost of implementing the Learning-Reference System 
will depend upon how elaborate the system is to be and how many 
users it is to serve simultaneously. One system might incorpor- 
ate all the major functions of the design but minimize the number 
of desirable-but-not-necessary services. Another system might 
include not only the major and secondary functions of the design 
but also automation of support functions such as supervising and 
updating. 

Since the possible configurations of computer facilities and 
versions of the Learning- Ref erenc© System are endless, the only 
way we can say something useful about time and cost figures is 
to take representative examples of a large system and a small one 
and try to make estimates for them: 



System I . Educational /training objectives primary 
, Limited version of sequence generator 
, One course availablve at a given time for the multiple 
purposes of 1 to 6 users 

. Small computer in the $50, 000-$70,00C- class with 
adequate time-sharing, display, and file management 
package ? 

Cost to Implement: $50-75,000 (direct labor cost) 

Time required: 12 months 

(Cost of course materials and computer time for system 
development not included.) 



System II . Information management as primary concern^ -training 
objectives, secondary 
. Ej^anded version of sequence generator 
. Up to six courses available at one time for the multi- 
ple pui^oses of 1 to 100 users 
. Large general purpose computer facility with adequate 
time- sharing, display and file management packages. 

Cost to implement: $100-150,000 (direct labor cost) 
Time required: 30 months 



(Cost of course materials and computer time for system 
development not included,) 

continued on next page 
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What is involved in the day-to-day operation of a Learning- 
Reference System? Although the number of support personnel will 
vary with the size and complexity of the installation, we can 
identify the functions that will have to be carried out to keep 
the system operating efficiently: 

* .njflnaging . deciding facility policies, assigning support person- 

# scheduling system operations (on-line time for primary 
and secondary users, and for maintenance technicians), estab- 
lishing priorities for support personnel 

• pyogrsinming . system modification, loading of new course mater- 
ials, correction of stored course materials, improvement of 
data co^ lection procedures , and so on 

. updating and documenting : insertion of all changes required by 

I’svislon of course ma'cerials and additions to system documenta- 
tion 

' supervising : deciding the kinds of guidance the sequence gen- 
erator IS to provide for each client, entering starting data 
for each client into the system, evaluating performance data 
in order to guide initial learners more appropriately (optional) 

. research: collection of performance data concerning system 
functions, information block use and user behavior patterns j 
evaluation of course materials; comparison of conditions; 
recommendation of changes. 

In a small educational facility these functions might be carried 
by a staff of two or three , while In a large facility each 
function might require several pecple, (Personnel required, for 

the preparation of course materials were discussed in an earlier 
block .) 



Operating 
the System 
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CHAPTER 9 FINAL CONSIDERATIONS AND SUMMARY 



OVERVIEW OF THIS CHAPTER 



Introduction The major features of the Learning-Reference System have been 

charted in fairly detailed form, but the plan is only the 
springboard for the future. Even as the first version evolved, 
ideas for later versions were being recorded. Several genera- 
tions of the system can already be dimly perceived. 

This In the next few maps, plans for later expansions will be 

Chapter mentioned. Finally we shall return to consider the immediate 

utility and importance of the present versionv and the research 
contributions to be expected from its implementation. 
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THE NEXT STEP 



Introduction 



Display 



O 




In the first version if this system, a number of desirable 
features were left out because we were concentrating on develop- 
ing the main mechanisms that would move the information blocks 
around and that would control interactions with the user in 
various ways. 

Some important features were left out also because their design 
depended too much on the capabilities of the specific conputer 
that may be involved. 

Now the possibilities for incorporating such features may be 
considered. Some have already been mentioned under ''Limita- 
tions of Hiis Version" (Chapter 1) as being desirable next 
steps , 



ITie most inportant addition to the system will be the programs 
that determine how display capabilities of the given conputer 
will be utilized in the Learning- Reference System, 

It would have a tremendous impact on the system, for instance, 
if the display scopes could be used as a response medium . 

If area or marginal displays could be generated independently 
of the main information block displays, then such aspects as 
these could be showm 

, possible next topics 

, permissible commands 

, local index for related topics 

, pronpting comments. 

Hie users could indicate many of their commands and selections 
by means of the light pen rather than by keyboard, Iliis would 
probably make it much easier for the user to learn to operate 
the system. 

The parameters of displays that can be manipulated during a 
session (brightness, size of characters, formats, etc,) will 
have to come under control of either the system or the user. 

New dimensions for controlling these will have to be defined 
and implemented in decision tables. 

Various display options can be used to increase the Impact 
of in for mat ion -block materials . It is the author or curriculum 
specialist who would calX upon these. For example, diagrams 
or sketches can appear in one area to illustrate the concepts 
or processes being ej^lained by the information block in 
another area. Tables and diagrams might be revealed only a 

continued on next pa 
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(continued) 

Display 



Sequencing 
and Control 
Decisions 



part at a time as the text explains each. Structures cm be 
"exploded" or "imploded" to show how parts fit together, and 
so on, Ihe whole area of graphics technology can be drawn upon 
for ways to improve displays, 

Hus role of animatio n as an educational tool has been explored 
in films and TV, but not enough is known about it yet in the 
con^uter setting. Animation techniques could be applied to 
highlight Important points, focus attention, illustrate move- 
ment in networks, processes and the like. Ihe impact of such 
techniques will undoubtedly vary with the individual differences 
in users and will have to be studied in controlled research. 

If con^uter-display capabilities are to be put to practical use 
in improving the communication value of instructional 
materials, then the system must be prepared with supporting 
programs that will enable authors and designers to use the^ 
dl splay op t i on s without the need for special training. 



Often through this report we have mentioned our intention to 
eliminate the supervisor from the system eventually. It is 
prdbabTy not wise to plan for this until after a period of 
es^erience with an operating system, IVhile many of the super- 
visor’s functions have already been marked for automation md 
could easily be turned over now, there are still other aspects 
of a new system taking to the air for the first time that 
require fecidback from a knowledgeable observer. Whether that 
person be a training supervisor or researcher is not in^ortant 
at the moment, but it does appear that a period of shakedown 
experience should precede sai atten^t to formulate the algo- 
richms to do the supervisor’s work. 

The expansion of dynamic sequencing decisions made during the 
sessions is another aim for future system versions, ^“e 
system could be made more responsive to the individual if It 
had more opportunities for reacting to user behavior patterns 
and performance data as each session unfolds. 

The definition of what aspects should be evaluated on a con- 
tinuing basis and how they should Influence sequencing decisions 
can also probably take place most efficiently after an opera- 
tional version of the system becomes available. Study of 
interaction records and real user data would help in evaluating 
the feasibility of various alternatives. 

At any rate, an on-going objective of the system should be to 
continue to in^rove its responsiveness to the Individuality 
of its users. 
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Con^uters commonly possess routine capabilities that the 
present version does not utilize. It has been our intention 
to make them a part of the services of the Leaming-Reference 
System but their inclusion must wait until a specific computer 
has been selected, 

Tlie remarkable power of the computer to perform on-line computa- 
tlons should be put at the disposal of the users of the 
Learning -Reference System. To be able to switch from an instruc- 
tional mode to a computing mode is an exciting possibility for 
those who are learning or consulting mathematical or engineering 
materials. The motivationa.l aspects alone of being able to do 
real work with a big computer system are incalculable. 

The power of a conputer with display capacity to aid in practical 
design problems - -whether it be circuitry design or ardii te ctur aX 
design — ^might also be available for the users of some systems. 
Dynamic display options plus the capacity to "draw" or "write" 
on the scopes could enable tlie users to create and ej^eriment in 
unpredictable ways. 

These features cannot be added to the design of our system in 
any useful way until more is known about the memory capacity 
and display capabilities of the specific computer it is to use. 

nie printout capabilities of a computer system can also be con- 
sidered an important educational resource. As we mentioned at 
the end of Chapter 1, there may be economical problems in 
controlling its use. Nevertheless, to allow system usors to 
take home materials for review or reference may actually result 
in on-line time savings. 

The Learning -Reference System will have to add the programs to 
control such printouts and to enable system clients to specify 
what they want and how it should be formatted, 

Conputers are able to tell time in remarkably small increments. 
This inherent capacity can be used in many ways which our 
system design does not include yet. The Briefing Mode, for 
instance, plans to adjust its presentations to clients according 
to the amount of time they want to spend. This needs implementa- 
tion. 

Evidence of varying quality suggests that time pressures may 
actually facilitate the learning of certain types of individuals. 
Time limitations on tests may be sometimes desirable as well. 

Time spent on meps, feedback questions and the like is useful 
evidence to guide revisions of materials. 

Later versions of the Leaming-Reference System, then, should 
include programs to deal with time controls and time recording. 



LATER VERSIONS 



Introduction Among our dreams of second or third generation Learning- 

Reference Systems, we see a growing sophistication in the 
inclusion of other media under computer control. We also see 
a more effective utilization of the associative capacities of 
the computer. 



Media Future versions should have richer data bases in the sense 

that many other media can be controlled and inserted into the 
user-system Interacticn by the sequence generator. Among these 
are ■ 

, printed materials ("hard copy," reproductions, etc.) 

, real objects (for illustration, identification, 
diagnoses, etc.) 

, physical models (of the brain, of the telephone, 
of the snail, etc.) 

, sketch pads 

, tape recordings 

, slide and film projections 

, video tapes. 

We suspect that an important part of the enjoyment of learning 
is the unexpected encounter with surprises and humor. We hope 
that future versions will include a liberal lacing of these 
Ingredients coupled with a subtle appreciation for the proper 
moment for Inserting them. 

It is unrealistic to imagine that course designers or authors 
will make full use of these system capabilities unless the 
system is ready with programs that will help them with very 
practical techniques for specifying what they want. 

The next version of the system will expand the cross-referencing 
procedures of the system. At present we have only detailed 
tables of contents and a cross-referenced index for each course. 
Greater power, especially for self-directed learners, can be 
gained by adding linkages across courses . Some provisions might 
be included to let users comment on some of their own insights 
about the links . 
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LATER VER: 



Simulation 

and 

Games 



IONS, continued 



TTie mechanisms of the Learning -Reference System are so flexible 
that games and simulation exercises can be made a part of the 
learning, referencing, reviewing services. 

Short courses to teach effective use of the system can be 
accomplished by games simulating system operations. Earlier 
we mentioned the desirability uf using simulation to start users 
on the path to self-directed learning. Short exercises, dis- 
cussions of useful strategies, and hypothetical problems can be 
confined to work toward that objective. 

To incorporate such possibilities in a learning system requires 
considerable work. The feasibility of adding them depends on 
the specific system's capacity and on the addition of aids to 
the game designers so that they can easily take advanta'ge of 
the full range of system capabilities. 



BLUE SKY VERSIONS 



Introduction Organized efforts to advance the efficiency of con^uter-aided 

Instruction differ in many ways, but one dimension along which 
they may be distinguished is that of the static or dynamic 
nature of the information base. 

The majority of systems have data bases whose structure is fixed 
in advance, everything having been entered by the authors, 
including canned messages and all the remedial branches that can 
be anticipated. An example is the Air Force's Computer-Directed 
Training System ("The Development of a Computer-Directed 
Training Subsystem and Computer Operator Training Material for 
the Air Force Phase II Base Level System," AD 702-529). 

A small but potentially very In^ortant line of development lies 
in the area of artificial intelligence. Exemplified by a 
system such as Carbonell and Collin's ("Mixed-Initiative Systems 
for Training and Decision-Aid Applications," ESD-TR- 70-373) , the 
data bases of these systems are capable of generating questions 
and answers not anticipated by the data base writer. 

Still in their infancy, these so-called generative systems 
function through executive programs that can analyze the seman- 
tic and syntactic structures of the stored information. They 
are "knowledgeable" in the sense that they can produce meaning- 
ful information not stored per se in computer memory, 

Ihe possibility of a future union of these diverse approaches 
intrigues us now. 



In this context our Learning-Reference System belongs in the 
class of static or structured data bases from which no new 
inferences or generalizations can enterge as a result of the 
master control program. 

The extensive nature of our underlying decision structures 
enables our system to control the conditions of user-system 
interaction, and to respond to the longer-term histories of 
sys-uem users. This capacity to respond individually to control 
conditions and to es^eriment with teaching strategies offers 
certain present advantages for practical applications md for 
controlled research into optimum conditions of learning — yet w# 
foresee the day when artificial intelligence capacities and 
systems such as ours may combine to give us an automated tutor 
whose memory, intelligence and capacity to disceiTi the 
individual needs of users far surpass the powers of human 
teadiers. 
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UNIQUE RESEARCH CAPABILITIES OF THE SYSTEM 



Quotation 



Advantages 

and 

Issues 



The most inportant inpaat on eduoaiion of aornputer tedhnotogy 
. . , wiVl probed ly be by aupp tying a toot for finding out what 
we are doing^ for turning aneodotat impreaaioniatia anawera 
into eoientifioatly t&atabte anes^ and ao turning what haa been 
atmoat purety at art into a reapeotdbte eaienae-~ond without 
etimtnating the artiatio aepeata either » Reaearoh in education^ 
advmaea in eduoationat imderstanding^ eduoatim as a behatiorat 
soienoe witt be^ I thinks the moat importcoit outaome, 

R, W, Gerard 

(Dean of the Graduate Division, 
University of California, Irvine) 
in Computers and Education, p. xlv, 
McGraw-HiU, 1967, 



Tlie Le am ing- Reference System as outlined here would constitute 
a significant research facility . Its unique capacity to manipu- 
late and control condif ions and, to keep records makes possible a 
rigorous researcli attack on the inportant unsolved problems of 
education and training. Conditions can be changed and running 
records kept not only for the user's response patterns and 
performance data but also for the frequency patterms of infor- 
mation-block usage and commrnd usage under each experimental 
condition. 

The issue of how much guidance should be given a learner has 
long concerned educators',’ Now, as we have seen, this one system 
can offer varying degrees of guid^ce ranging all the way from 
total control of the leamer to no guidance of him at all. 
Research with eadi leamer using the system under systematically 
varied control conditions would begin to give us some long- 
sought answers. 

But nov; the issue becoines one not so much of how much to guide 
the learner but rather how much to guide what kinds of learners. 
Recent concem with fitting instruction to the individual student 
points up the need for studying the effect of guidance conditions 
on various kinds of users, distinguished on the basis of patterns 
of data stored in their data bases. 

Quite apart from the issue of how much control, if any, is best 
for each individual, there is the problem of prescribing infor- 
mation sequences and response requirements for the various 
learners or reviewers who are being in some degree guided by 
the system. In earlier discussions of this topic in Chapters 3 
and 4, prescription-making was largely left on the doorstep of 
the curriculum specialist or author with the assimption that 
he would know not only what sorts of sequences should be 

continued on next page 






UNIQUE RESEARCH CAPABILITIES OF THE SYSTEM, continusd 



(continued) assembled for which kinds of people, but also what kinds of reme- 
Advantages dial sequences are suitable to each. And we provided him with all 

and sorts of new and old information about each user so that he would 

*^sues have concrete data for making his decision about sequences. 

As a matter of fact, no one really knows very much about 
prescribing for the individual nor about what factors in the 
short- or long-term history of the person are relevant to 
sudi decisions. 

One of the most important advantages of the Learning-Reference 
System is that it is designed to find answers to these problems. 

It is structured so that sequencing decisions can draw on 
many aspects of the user’s background, interests, objectives, 
changing behavior within the session, and so on. Tills capabil- 
ity, coupled with its control over conditions and its record- 
keeping advantages, offers unprecedented research tools for 
"learning about learning." 

Ifnile a major research effort on individualization is a pressing 
need at the present time, there remain a nuniei' of oth.-r or 
related research problems which the Learning-Reference System 
' equipped to tackle: controlled comparisons of teaching 
strategies, the effect of feedback variations, motivational 
effects of different learning conditions, curriculum- mater la Is 
research, multi-media research, con^arlson of strategies for 
improving self-direction, and so on. 

The effect of various display paramete ‘:.j lOt only on learning 
but on reference and reviewing as well is another fertile 
research area that can now be explored lii relation to indivi- 
dual differences. 

While some of the richest res ear da problems center around 
initial learning, the related topic of reviewing and, by in^li- 
cation, retention, offers some interesting researdi possibili- 
ties--the effects of Individualized treatment here also, to 
mention only one. 

In tenns of long-run impact, probably the most important aspect 
of the Leaming-Reference System lies in its potential for 
searching out the wiswers to the practical problems of making 
information services more effective. 
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IMMEDIATE PRACTICAL USE OF THE SYSTEM 



Introductl 



Present 

Outcomes 



The present system marks an important step toward making the 
computer a practical force In education and training. 

It can guide the user or free him to use the information base 
as he wishes . 

It can deliver sequences for him that take into account his 
short-term and long-term history and that reflect a given 
teaching strategy. 

It can make available to him the facilities of a major compu- 
ting system--to compute, to design, to cross-link, to create, 
to learn, to browse, to experiment, to ej^lore. 



Immediate practical outcomes for tj’aining and job aids can be 
ej^ected from the implementation of this system plan on an 
existing computer--yet implementation is also Important for 
the further evolution and refinement of this system. It is 
only by practical experience that the validity of certain solu- 
tions and the feasibility of some features can be tested, 

Tnrough the experience of operating sudi a system and of 
receiving feedback from its users, more subtle decision lules 
can be formulated. 

Systematic observation of system users and study of their data 
and comments can lead to the devising of more useful aids for 
referencers, reviewers, and other users whose needs are not so 
clearly recopilzed now* 

CyCies of developmental testing and revision are required to 
Improve not only the information materials of the system but 
the operation of system functions as well, Hiese are the 
necassary sequels of implementation on an operating system. 



SUMMARY 



The computer-based system we have designed will provide 
Information services for a variety of users who need access 
to textual materials* learners, reviewers, referencers, 
browsers, and so on. iTie quick retrieval of relevant infor- 
mation is achieved by having materials sorted into blocks and 
labelled according to the class of information concerned. 
Information Mapping is the nMie given to the unique classifi- 
cation scheme that guides this basic organization of informa- 
tion j it also includes formatting rules and guidelines for 
effective display. 

The computer- control led system is so designed that users can 
explore the information base of organized, labelled blocks 
with the same freedom they exercise in a library. But for 
those who need guidance, the system can enter into the process 
of selecting and assembling those blocks that are relevant to 
the user’s objective. 

The system's role in information sequencing can vary from minor 
prompting all the way to total control of pacing a user through 
a subject matter to achieve a specific instructional goal. 

Whenever the system enters into the process of assembling 
information for displry, it is set up to take into account the 
individual user's needs, capabilities, interests and the like. 

While it is now possible to make reasonable prescriptions' for 
information sequences suited to certain individual differences, 
(a neophyte in mathematics will obviously not be shown the same 
displays as the college senior math major), there remain many 
gaps where research evidence in education and psychology simply 
does not exist to guide sequencing prescriptions. 

The present system is designed with the flexibility not only to 
Incorporate new research findings into the decision-making 
mechanisms but also to take an active role in gathering under 
controlled conditions the con^arative data that will extend our 
knowledge of how to make information presentations more 
effective for each Individual. 
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APPENDIX A 



Introduction ITie Inforination Map Classification Chart (in updated form) is 

reproduced here from a previous report. Information Mapping for 
Learning and Referen '^r<. Horn, Nicol, Kleinman and Grace, 
ESD-TR-69=296. 



MAP CLASSIFICATION CHART 




Types of Maps 


Description 


^ u — ^ 

Information Blocks 


Concepts 


A concept may be a: 

. technical term 

. generalization sentence 
. property sentence 
. rule sentence 
. relationship sentence 


. name of the concept 
. definition or description 
. theorem (or generalization) 
. formula 
. use 
. rule 
. example 
. non-example 
. introduction 
. synonym 
. notation 
. diagram 
. comment 
. analogy 
. related maps 


Structures 


A structure is: 

, a physical thing, or 
. something which can be 
divided into parts which 
have boundaries. 


. name of structure 
, meaning 
. function 

. /all of the concept blocks/ 
. parts and subparts 


Processes 


A process is some structure 
changing through time. The 
description of a process 
Involves writing about what 
happens during successive 
stages of time. 


. name of process 

, /all of concept blocks/ 

. /all of structure blocks/ 

. stage table 
. parts -function table 
. cycle chart 
. input -output table 
. cause-effect table 
. block diagram 
. PERT chart 

. WHIP chart 

, state table 
. condition 
, cause 
, effect 



continued on next page 
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MAP CLASSIFICATION CHART, continued 



Types of Maps 


Description 


Information Blocks 

f 


Procedures 


A procedure is a set of 
steps performed to 
obtain some specified 
■outcome . 


. name of procedure 
. /all of concept blocks/ 
. procedure table 
. flowchart 

. occasion for starting 
. when to stop 
, decision table 
. check list 
. work sheet 


Classifi- 

cations 


Classification is the 
sorting of things by 
concepts into categor- 
ies by the use of one 
or more sorting factors 
(criteria) . 


. classification table 
. classification sheet 
. classification list 
. outline 


Facts 


Facts are sentences 
about things done, 
things that are or were 
in existence, events, 
conditions, and so on, 
and are presented 
without supporting 
evidence . 


- s tatemant of fact 


Proofs 


Proofs are generally 
used in mathematical 
subjects for more dif- 
ficult theorems. 


. name of proof 
. assumptions 
. to prove 
. statement 
. reason 
. example 
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APPENDIX B 

EXAMPLE OF A PREREQUISITE CHART * 



Introduction Before writing begins, the complete prerequisite chart (Inclu- 

ding the special learning materials) is drawn up. This chart 
functions as a road map for the writer, showing him the nature 
of his task and the character of the terrain that lies ahead. 

But the chart is not only important as a guide to the writer- - 
it can also give the learner a clear idea of his learning task 
and of the direction the map presentations are neading. There- 
fore, prerequisite charts are included in Information map books. 

Example The arrows ai'e used to indicate the logical structure of the 

subject matter, i.e., the relationships among the concepts. 

The numbers refer to the teaching strategy or the sequence 
in which the topics are presented. The special learning and 
reference maps are shown in boxes . 



Tw© Ways QfJ 
Sp#©ifyingy 



Pr©vi«w (p 
of S€t 
Thtoiy 



Sfts 



ilfBtiiti 

of ft Set 




Venn PiegraK 
/ Stiisets 




Hull tek 



Set 




tvery ii 
m Snfeiit of. 

Ititlf Listing All @ 

iubsets of A 



Hie Null Set 
is m Subset of 
Eyefy ^t 



Set 



Coieiting 
Ail Si&iets 
of s Set 



Review 

Questiofii 



Review^® 

^listioii^ 



Review iQ 
Review 



Co^lei^nt 

Opereti^i Q 
.with ‘No or 
More Sets 

union 




Interieotion 



Differenee 




disjoint 

^ts 



FondBf Uhionj @ 
iro m Mtm vitli 
Different Mrabei^hipi 



Smery ® 




tovlM @ 


of ^t tiieoiy 




Quest iiw 



Union of 
3 or 



■ore sets 



CoapexiBi @ 
union end 
Inte^eetion 



Foraing Interseetions Q 




Review Q 


iron ^ts with Diffemit 




Meifaerships 




Qi^stions 



Inte^ootiOfi of 
3 or ■ore sets 




* This and the following three maps are reproduced from: Infonnation 
for Learning and Reference. Horn et al., 1969, ESD-TR-69-I9I1 
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EXAMPLE OF A PROCEDURE MAP 



Introduction This map shows the general case and the vorked exan^le side 

by side. 



HOW TO COMPUTI THE STANDARD DEVIATION 



r* 

! A set of n n^fbtri 

V 
E 
N 


EXAMPLE. 

Find the stvidard devis^an of the fol- 
lowing test scores: 

§6, 82 , 68, S3, 77, SS, as, PS, 81, 71 


STEP 


PROCEDURE 


EKAWLi 


1 


Add the given ms^ers to 
get the %\m’ 

£Xi 


The siai of *he ten test scotes Is 

EXj« ■ 100 


2 


Squere the sur and then 
divide by the ntnber of 
eaies to get: 

CEX^)‘/n 


tEXj)’/n ■ {800)*/10 ■ 640000/10 
■ 64000 


3 


Add the squares of the 
given nuAers to fom 

EX.» 


The %\m of the squares of the test scores 
is 

EX^^- 86^*82^*^'**71^ • 65234 


4 


StAtract the result of 
Step 2 : 1 the result 

of Step 3: 

EXj* - (Ex^)Vn 


EXj* - (IXi)Vn ■ 65234 - 64000- 1234 


S 


Divide the result of Step 
4 by the nuriber of oaiei 
minus one to get : 

_ EXi* - (EXi)Vn 
n “ 1 


S* - ■ 137.1111 

s 


6 


The standard deviation is the 
square root of the result of 
Step 5: 

S« ✓ S* 


S ■ 1^137.1111 ■ 11,71 
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EXAMPLE OF A COMPARE -AND-CONTRAST TABLE 



Introduction 



Example 



o 




We found that the distinction between the concepts of 
union and intersection was frequently missed by beginning 
students of set theory. Thus, the following map was written. 



r" 



aid 



CoMPARiNri, , , Intersection AND. . . 



Union 



Symbol 

DefiniEion 



Vonn Diagram 



Exa^le One 



Two 



n 



Th# intarseQEion of two sati 
F and Q is the saE of all 
members belonging to both 
F and Q. 



Tha shaded part ii A ^ B 





u 























H s { 9, li, S3 ) 

N * { 3, i, 7, 9 > 

Notica this intarsaotion , 

M O N = { 5, 9 } 



« f I, 2 3 ) 

^ \ 4 j I i ^ ) 

Notica this intarsacticni 

E n F - » 

(Thera are no conon 
alemants] 



u 



The union of two sets P and Q 
is the set of all elements that 
are members of P or Q or both. 



The shaded part is A U B- 



M = { S, 9, IS, IS } 

N . ( 3, S, 7, 9 ) 

.Notice this u.non, 

M ^ N - ( 3, i, 7, 9, IS, 33 



= { 1 , 2 , 3 } 

* { 4, S, 6 } 

Notice this imion, 

E y P = ( 1, 2, 3, 4, I, a 



Rf lated 
Pagas 



elementij 4 
union, 30 



intersection p 31 



set i 4 
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EXAMPLE OF A SUMMARY MAP 



Introduction 



Example 



This table was designed to tie up the topics related to 
conditional probability. Readers who are not familiar with 
the subject matter can nevertheless note several Important 
features of the example. The introduction underlines the 
importance of the topic and suggests to the student the nature 
of his learning task. In the left coliamn simple questions are 
paired with concrete examples to help the student remember 
the distinctions. Formulas are as usual accompanied by 
verbal descriptions . 



POSTVIEW OF CCNOITIONAL PROBABIirTY 



introduction f Conditional probability is one of the most useful tools in pro- 

ibability theo^. We have seen how the equation can be 
rearranged to give the multiplication theorem and the indepen- 
! dence definition, both important concepts. 

! The inter- re Utedness of these ideas is a great boon to memory. 

If the conditional probability definition is thoroughly under- 
. stood and stored in memory, it can serve as the key to unlock 
1 recol lection of how the other formulas can be derived. 

For review, we re-state the definition of eonditional probabiU 
j ity along with the concepts derived from it. 


i QUESTION 


CONCEPT AND FORMULA 


What is the probability 
of A, given that B has 
occurreJ? 

[Given a red-haired person, 
what is the probability 
that he has blue eyes?] 


Conditional Probability: 


pcAii) = 


the conditional "the probability that both^ 
probability of = events will occur divided 
A, given B by the probability of 1 


What is the probability 
that both A and B will 
occur? 

(What are the chances 
of winning Olympic 
medals in both track 
and iwimming?) 


Multiplication Theorem^ 


P(AnB) - P(B)-PCAlB) 


'"the probability" ^he probability of B time? 

that both events = the probability of A, 
occur given 1 


Are the events 
A and B Independent? 

(Is the event 
flunking math ' indepen- 
dent of the 
event 

'"flunking hlitor>"?‘*] 


Independence Definition: 

Two events are Independent if and only if: 
P(AnB) « P(A)-P(B) 


'the probability^ Ihe product of thT^ 

that both events ^ separate probabilities, 
occur 


(liven two independent 
events A and 1, what 
is the probability \ 

that both of them occur? 

i |lf Tom and 1 both roll 
.a die, what is the pro- 
Ibability that we both 
j get s ixes?] 


^fultipl ication Rule for Independent Events: 


[This is the independence equation Just above*] 
* 
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APPENDIX C 

EXAMPLE OF A CONCEPT MAP 
DIVISIONS OF A COBOL SOURCE PROGRAM* 



Introduction In leaving detailed instructions for anyone about carrying out 

a conplex task, we commonly give them several types of 
information: we identify the task and ourselves first of all, 
then we say what pieces of equipment are to be used and how 
they are to be set up, and then we specify the kinds of 
materials that are to be processed or produced, and finally we 
explain the procedures to be followed. 

The conputer too needs these same kinds of information in 
order to perform its tasks. The COBOL source program is 
organized to specify such instructions in a systematic way. 

Description A source program in COBOL has four divisions: 



the Identification Division , stating the name of the 
program, its author, and probably the date 

» e, the Environmeiyt Div ision , telling about the equipment 

to be used, such as' the specific conputer (if the 
facility has more than one) and the various input- 
output devices 



♦ the Data Division , explaining what data 5. terns are 
going to be used and how they are arranged 

. the Procedure Division , setting out the steps that 
are to be followed in processing the data. 

In a COBOL source program these four divisions appear in 
the order given here. 



Comment The Data Division will be described next so that we may see 

first what sorts of "ingredients" are going to be worked on 
in the Procedure Division, 



In actual progran writing, this is the order one would 
ordinarily follow. It is time enough to add identification 
and equipment information after the main idea has been worked 
out about how the procedures are to be applied to the data. 



* COBOL , Cambridge, Mass., Information Resources, 
cited on p. 12, 
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A new conception of coraputer-based instructional systems is presented in this design 
of a system that can deliver individualized infonnation sequences not only to 
learners and trainees, but to reference workers, reviewers, browsers and the like. 
Underlying the system is a flexible data base, organized into labelled, movable 
information blocks according to the principles of Information Mapplng--a system for 
categorizing and displaying information. Hiis report is itself written In modified 
Infopnatlon-Mapping style. A significant feature of this computerized information 
seryice is that the control over information selection and arrangement can be 
assipied entirely to the user, entirely to the system, or to both In one of many 
possible patterns of shared responsibility. When the system takes part in informa- 
tion-sequencing decisions, its many mechanisms for individualizing come into play. 
The executive program consults short-term and long-term^data about the individual, 
his objectives, capabilities. Interests and present status before it selects and 
arranges blocks from the data base to display for him. Evaluation and feedback 
provisions are also individualized, nie system's capability for controlling condi- 
tions and recording user-system interactions will mak.e it a" valuable force in 
research on individualization in training and education, TTie development of this 
complex design was facilitated by a. Document at ion -Updating System that produced 
system documents in Information-Mapped form and kept them up to date throughout 
the project. 
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